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Basic Elements of Statistical Decision Theory and Statistical Learning Theory 

This paper reviews and contrasts the basic elements of statistical decision theory and 
statistical learning theory. It is not intended to be a comprehensive treatment of either 
subject, but rather just enough to draw comparisons between the two. 


Throughout this module, let X denote the input to a decision-making process and Y 
denote the correct response or output (e.g., the value of a parameter, the label of a 
class, the signal of interest). We assume that X and Y are random variables or random 
vectors with joint distribution Px y (x, y), where x and y denote specific values that 
may be taken by the random variables X and Y, respectively. The observation X is 
used to make decisions pertaining to the quantity of interest. For the purposes of 
illustration, we will focus on the task of determining the value of the quantity of 
interest. A decision rule for this task is a function f that takes the observation X as 
input and outputs a prediction of the quantity Y. We denote a decision rule by Y or 
f(X), when we wish to indicate explicitly the dependence of the decision rule on the 
observation. We will examine techniques for designing decision rules and for analyzing 
their performance. 


Measuring Decision Accuracy: Loss and Risk Functions 


The accuracy of a decision is measured with a loss function. For example, if our goal is 
to determine the value of Y, then a loss function takes as inputs the true value Y and the 
predicted value (the decision) Y = f (X) and outputs a non-negative real number (the 
“loss”) reflective of the accuracy of the decision. Two of the most commonly 
encountered loss functions include: 


1. 0/1 loss: £9 /4 (¥, Y) = I,.,, which is the indicator function taking the value 


YY’ 
of 1 when Y # Y and taking the value 0 when Y (X) = Y. 
— — 2 
2. squared error loss: £2 (¥, Y) = || Y —Y ||,, which is simply the sum of 


squared differences between the elements of Y and Y. 


The 0/1 loss is commonly used in detection and classification problems, and the squared 
error loss is more appropriate for problems involving the estimation of a continuous 
parameter. Note that since the inputs to the loss function may be random variables, so is 
the loss. 


A risk R(f) is a function of the decision rule f, and is defined to be the expectation of a 
loss with respect to the joint distribution Px y (a, y). For example, the expected 0/1 
loss produces the probability of error risk function; i.e., a simply calculation shows 


that Rost (f) = El(Ipayzy] = Pr (f (X) # Y). The expected squared error loss 
produces the mean squared error MSE risk function, R2 (f) = E/|| f (xX) —Y 2] ; 


Optimal decisions are obtained by choosing a decision rule f that minimizes the desired 
risk function. Given complete knowledge of the probability distributions involved (e.g., 
Px y (x, y)) one can explicitly or numerically design an optimal decision rule, denoted 
f’, that minimizes the risk function. 


The Maximum Likelihood Principle 


The conditional distribution of the observation X given the quantity of interest Y is 
denoted by Pxjy (x|y). The conditional distribution Py|y (|y) can be viewed as a 
generative model, probabilistically describing the observations resulting from a given 
value, y, of the quantity of interest. For example, if y is the value of a parameter, the 
Pxjy (2|y) is the probability distribution of the observation X when the parameter 
value is set to y. If X is a continuous random variable with conditional density 

px\y (x|y) ora discrete random variable with conditional probability mass function 
(pmf) px\y (a|y), then given a value y we can assess the probability of a particular 
measurment value y by the magnitude of either the conditional density or pmf. 


In decision making problems, we know the value of the observation, but do not know 
the value y. Therefore, it is appealing to consider the conditional density or pmf as a 
function of the unknown values y, with X fixed at its observed value. The resulting 
function is called the likelihood function. As the name suggests, values of y where the 
likelihood function is largest are intuitively reasonable indicators of the true value of the 
unknown quantity, which we will denote by y . The rationale for this is that these 
values would produce conditional densities or pmfs that place high probability on the 
observation X = z. 


The Maximum Likelihood Estimator (MLE) is defined to be the value of y that 
maximizes the likelihood function; i.e., in the continuous case 
Equation: 


y(X) = argmax Px|y (X|y) 


with an analogous definition for the discrete case by replacing the conditional density 
with the conditional pmf. The decision rule g (X) is called an “estimator,” which is 
common in decision problems involving a continuous parameter. Note that maximizing 
the likelihood function is equivalent to minimizing the negative log-likelihood function 


(since the logarithm is a monotonic transformation). Now let y denote the true value of 
Y. Then we can view the negative log-likelihood as a loss function 
Equation: 


a; (v,9") = — log pxjy (X|y) 


where the dependence on y" on the right hand side is embodied in the observation X on 
the left. An interesting special case of the MLE results when the conditional density 
Pxjy (X|y) is a Gaussian, in which case the negative log-likelihood corresponds to a 
squared error loss function. 


Now let us consider the expectation of this loss, with respect to the conditional 
distribution Pyjy (X | ee 
Equation: 


—Ellog pxy (X|y)| = / log (“ap )e (2|y") ae 


The true value y minimizes the expected negative log-likelihood (or, equivalently, 
maximizes the expected log-likelihood ). To see this, compare the expected log- 


likelihood of y with that of any other value y: 
pxiv (X|y’) 
log | ————_—— 
Px\y (X|y) 


Equation: 
PX|Y (x|y") * : 
= ] ——____ d 
/ . ( Pxiy (z\y) Jews (z|y af 


= KL(pxy (2|y'),pxy (ely) 


E Hlog Pxiy (x v’) — log pxyy (X DD) = # 


The quantity KL (p X|Y (x ly’) »PX\|Y (aly)) is called the Kullback-Leibler (KL) 
divergence between the conditional density function p x\y (x|y") and pxy (aly). The 


KL divergence is non-negative, and zero if and only if the two densities are equal [link]. 
So, we see that the KL divergence acts as a sort of risk function in the context of 
Maximum Likelihood Estimation. 


The Cramer-Rao Lower Bound 


The MLE is based on finding the value for Y that maximizes the likelihood function. 
Intuitively, if the maximum point is very distinct, say a well isolated peak in the 
likelihood function, then the easier it will be to distinguish the MLE from alternative 
decisions. Consider the case in which Y is a scalar quantity. The “peakiness” of the log- 
7 logp xy (x|y) 

ar , at the 
point of maximum likelihood. The higher the curvature, the more peaky is the behavior 
of the likelihood function at the maximum point. Of course, we hope that the MLE will 
be a good predictor (decision) for the unknown true value y . So, rather than looking at 
the curvature of the log-likelihood function at the maximum likelihood point, a more 
appropriate measure of how easily it will be to distinguish y from the alternatives is 
the expected curvature of the log-likelihood function evaluated at the value y . The 
expectation taken over all possible observations with respect to the conditional density 


* * 2lo x 
Px\y (xy ). This quantity, denoted I (y ) = [- Sear | | . is called the 
y=Yy 


likelihood function can be gauged by examining its curvature, — 


Fisher Information (FI). In fact, the FI provides us with an important performance 
bound known as the Cramer-Rao Lower Bound (CRLB). 


The CRLB states that under some mild regularity assumptions about the conditional 
density function px\y (a|y), the variance of any unbiased estimator is bounded from 


below by the inverse of the J(y’) [link], [link], [link]. Recall that an unbiased estimator 


is any estimator Y that satisfies E ba = y’. The CRLB tells us is that 


Equation: 


If Y is a vector-valued quantity, then the expected negative Hessian matrix (matrix of 
partial second derivatives) of the log-likelihood function is called the Fisher 
Information Matrix (FIM), and a similar inequality tells us that the variance of each 
component of any unbiased estimator of y” is bounded below by the corresponding 
diagonal element of the inverse of the FIM. Since the MSE of an unbiased estimator is 
equal to its variance, we see that the CRLB provides a very useful lower bound on the 
best MSE performance that we can hope to achieve. Thus, the CRLB is often used as a 
comparison point for evaluating estimators. It may or may not be possible to achieve 
the CRLB, but if we find a decision rule that does, we know that it also minimizes the 
MSE risk among all possible unbiased estimators. In general, it may be difficult to 
compute the CRLB, but in certain important cases it is possible to find closed-form or 
computational solutions. 


Bayesian Decision Theory 


Bayesian Decision Theory provides a formal system for integrating prior knowledge 
and observed observations. For the purposes of illustration we will focus on problems 
involving continuous variables and observations, but extensions to discrete cases are 
straightforward (simple replace probability densities with probability mass functions, 
and integrals with summations). The key elements of Bayesian methods are: 


1. a prior probability density function py (y) describing a priori knowledge of 
probable states for the quantity Y; 
2. the likelihood function p x,y (z|y), as described above; 


3. the posterior density function py|x (y|x). 


The posterior density is a function of the prior and likelihood, obtained according to 
Bayes rule: 
Equation: 
Px\y (z|y)py (y) 
J pxy (zly)py (y)dy 


Py|x (y|z) = 


The posterior is an indicator of probable values for Y, based on the prior knowledge 
and the observation. Several options exist for deriving a specific estimate of Y using the 
posterior. The mean value of the posterior density is one common choice (commonly 
called the posterior mean). The posterior mean is the decision rule that minimizes the 
expected squared error loss (MSE risk) function. The value y where the posterior 
density is maximized is another popular estimator (commonly called the Maximum A 
Posteriori (MAP) estimator). Note that the denominator of the posterior is independent 
of y, so the MAP estimator is simply the maximizer of the product of the likelihood and 
the prior. Therefore, if the prior is a constant function, the MAP estimator and MLE 
coincide. 


Statistical Learning 


In all of the methods described above, we assumed some amount of knowledge about 
the distributions of the observation X and quantity of interest Y. Such knowledge can 
come from a careful analysis of the physical characteristics of the problem at hand, or it 
can be gleaned from previous experience. However, there are situations where it is 
difficult to model the physics of the problem and we may not have enough experience 
to develop complete and accurate probability models. In such cases, it is natural to 
adopt a statistical learning approach [link], [link]. 


Statistical learning methods are based on developing decision rules or estimators based 
only on a collection of training examples, rather than predetermined probability models. 


Statistical learning methods are often said to be distribution-free, since they do not 
assume particular probability models. The canonical set-up for statistical learning is as 
follows. We begin with a collection of training examples, {(X;, Y;)};_,, which are 
assumed to be independently and identically distributed according to an unknown 
probability distribution Px y (x, y). If we knew Px y (a, y), then we could compute a 
desired risk function and design an optimal decision rule using the methods described 
above. In essence, the training examples give us a glimpse at the underlying 
distribution, but our knowledge of it is far from complete. We cannot exactly compute a 
risk function, and therefore we cannot derive a corresponding optimal decision rule. 


There are at least two ways to proceed at this point. One possibility is to use the training 
examples to estimate the joint probability distribution, and then use this estimate to 
derive an decision rule. Unfortunately, the (general-purpose) problem of estimating a 
distribution is often more difficult from a limited pool of data than is the problem of 
designing a specific-purpose decision rule. For this reason, a second possibility is more 
commonly favored in practice. Rather than estimating the complete distribution, one 
can use the training examples to directly design a decision rule. More precisely, perhaps 
the most common approach is to use the training examples to compute an estimate of 
the desired risk function. 


Suppose that we are interested in minimizing a particular risk function. Recall that the 
risk is the expected value of a chosen loss function. Let £ (2 Y) denote the loss, and 
let f(X) denote a candidate decision function, mapping observations to predictions 


about Y (ie., Y = f (X)). The empirical risk function is constructed from the 
training examples as follows: 
Equation: 


RU) = Ses (X),¥9). 


This is simply the average loss of the decision rule f over the set of training examples. 
Note that since the training examples are independent and identically distributed, the 
expected value of the empirical risk is equal to the true risk R(f) = E/é(f(X), Y)]. 
Moreover, we known (according to the law of large numbers) that the empirical risk 
tends to the true risk as the size of the training sample increases. These facts lend 
support to the idea of choosing a decision rule to minimize the empirical risk. 


Empirical risk minimization (ERM) is just this process. Given a collection of possible 
decision rules, say “, ERM selects a decision rule according to 
Equation: 


fn = in R(f). 
f argmin (f) 


The selected rule, fa, obviously depends on the given set of training examples, and 


therefore it is itself a random quantity. The theoretically optimal counterpart to f,, is the 
decision rule that minimizes the true risk 
Equation: 


7 = argmin R (f). 


The central problem in statistical learning is to quantify how close fn performs relative 
to f . Note that R ( f *) << (in), since f minimizes the true risk. Thus, one way to 


fay , x. 7 daze 
gauge the performance of f,, relative to fis to show that there exists small positive 
values € and 6 such that with probability at least 1 — 6 we have 
Equation: 


If an inequality of this form holds, then we say that fr is a Probability Approximately 
Correct (PAC) decision rule [link]. 


To show that the empirical risk minimizer is a PAC decision rule, we first must 
understand how closely the empirical risk matches the true risk. First, let us consider the 
empirical and true risk of the decision rule f. Assume that the loss function is bounded 
between 0 and 1 (possibly after a suitable normalization). Then the empirical risk 
function is a sum of independent random variables bounded between 0 and 1. 
Hoeffding's inequality is a bound on the deviations of such random sums from their 
corresponding mean values [link]. In this case, the mean value is the true risk of f, and 
Hoeffding's inequality states that 

Equation: 


P(\R(f)— R(f)|>6) < 26". 


Another equivalent statement is that the inequality IR (f) — R(f)| < € holds with 


probability at least 1 — 2e2"©’ Thus, the two risks are probably close together, and the 
greater the number of training examples, n, the closer they are. 


Now we would like a similar condition to hold for all f € ¥, since ERM optimizes 
over the entire collection ¥. Suppose that F is a finite collection of decision rules. Let 
|.F| denote the number of rules in ¥. The probability that the difference between the 
true and empirical risks, of one or more of the decision rules, exceeds € is bounded by 
the sum of the probabilities of each individual event of the form IR (f) -R(f)| >6 


the so-called Union of Events bound. Therefore, with probability at least 
1 — |F|2e-?”" we have that 
Equation: 


IR(f) -R(A)| <e 


for all f € F. Equivalently, setting 6 = 2|.F lene we have that with probability at 
least 1 — 6 and forall f Ec F 


Equation: 
R(f) — Ri <j BF ee) 


Notice that the two risks are uniformly close together, and the closeness indicated by 
the bound increases as n increases and decreases as the number of decision rules in F 
increases. In fact, the bound scales with log |.¥|, and so it is reasonable to interpret the 
logarithm of the number of decision rules under consideration as a measure of the 
complexity of the class. 


Now using this bound, we can show that So is a PAC decision rule as follows. Note that 
with probability at least 1 — 6 


Equation: 
AG) Rp) a eee 
< R(f") oq) SEE) 
< K(f’) +2yf 25 lF I+ tos /8) 


where the first inequality follows since the true and empirical risks are close for all 
f € F, and in particular for his the second inequality holds since by definition f,, 


minimizes the empirical risk, and the third inequality holds again since the empirical 
risk is close to the true risk for all f, in this case for fin particular. So, we have shown 


that f,, is PAC. 


PAC bounds of this form can be extended in many directions, for example to infinitely 
large or uncountable classes of decision rules, but the basic ingredients of the theory are 
essentially like those demonstrated above. The bottom line is that empirical risk 
minimization is a reasonable approach, provided one has access to a sufficient number 
of training examples and the number, or more generally the complexity, of the class of 
decision rules under consideration is not too great. 


Further reading 


Excellent treatments of classical decision and estimation theory can be found in a 
number of textbooks [link], [link], [link], [link]. For references on statistical learning 
theory, outstanding textbooks are also available [link], [link], [link] for further reading. 


Elements of Statistical Learning Theory 


Three Elements of Statistical Data Analysis 


e 1. Probabilistic Formulationof learning from data and prediction 
problems. 
e 2. Performance Characterization: 


concentration inequalities 
uniform deviation bounds 
approximation theory 
rates of convergence 


o Oo 0 90 


e 3. Practical Algorithmsthat run in polynomial time (e.g., decision 
trees, wavelet methods, support vector machines). 


Learning from Data 


To formulate the basic learning from data problem, we must specify several 
basic elements: data spaces, probability measures, loss functions, and 
Statistical risk. 


Data Spaces 


Learning from data begins with a specification of two spaces: 
Equation: 


X = Input Space 
Equation: 
Y = Output Space. 


The input space is also sometimes called the “feature space” or “signal 
domain.” The output space is also called the “class label space,” “outcome 
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space,” “response space,” or “signal range.” 


Example: 
Equation: 


% =R? d-dimensional Euclidean space of ° feature vectors”’ 
Equation: 


Y = {0,1} two classes or * ‘class labels’’ 


Example: 
Equation: 


2 =R_ one-dimensional signal domain (e.g., time-domain) 
Equation: 


Y=R real-valued signal 


A classic example is estimating a signal f in noise: 
Equation: 


Y=f(X)+W 


where X is arandom sample point on the real line and W is a noise 
independent of X. 


Probability Measure and Expectation 


Define a joint probability distribution on 2 x Y denoted Px y. Let 

(X, Y) denote a pair of random variables distributed according to Px y. 
We will also have use for marginal and conditional distributions. Let Px 
denote the marginal distribution on X, and let Py|x denote the conditional 
distribution of Y given X. For any distribution P, let p denote its density 
function with respect to the corresponding dominating measure; e.g., 
Lebesgue measure for continuous random variables or counting measure 
for discrete random variables. 


Define the expectation operator: 
Equation: 


Exy [f(X,Y)] = / i eareea= / pase, pdeey. 


We will also make use of corresponding marginal and conditional 
expectations such as fy and Fy\x. 


Wherever convenient and obvious based on context, we may drop the 
subscripts (e.g., instead of &'x y) for notational ease. 


Loss Functions 


A loss function is a mapping 
Equation: 


L:YxXBMoR. 


Example: 

In binary classification problems, ¥ = {0,1}. The 0/1 loss function is 
usually used: £ (yi, y2) = 1y,4y., where 1, is the indicator function which 
takes a value of 1 if condition A is true and zero otherwise. We typically 


will compare a true label y with a prediction %, in which case the 0/1 loss 
simply counts misclassifications. 


Example: 

In regression or estimation problems, 2“ = R. The squared error loss 
function is often employed: @ (yi, y2) = (yi — y2)2, the square of the 
difference between y; and yo. In application, we are interested in a true 
value y in comparison to an estimate y. 


Statistical Risk 


The basic problem in learning is to determine a mapping f : 2% +> Y that 
takes an input x € & and predicts the corresponding output y © Y. The 
performance of a given map f is measured by its expected loss or risk: 
Equation: 


R(f) = Exyle(f(X), Y)I. 


The risk tells us how well, on average, the predictor f performs with respect 
to the chosen loss function. A key quantity of interest is the mininum risk 
value, defined as 

Equation: 


R =inf R(f) 
where the infinum is taking over all measurable functions. 


The Learning Problem 


Suppose that (X,Y) are distributed according to Px y ((X,Y) ~ Px y for 
short). Our goal is to find a map so that f(X) ~ Y with high probability. 
Ideally, we would chose f to minimize the risk R(f) = E/é(f(X), Y)). 
However, in order to compute the risk (and hence optimize it) we need to 
know the joint distribution Px y. In many problems of practical interest, the 
joint distribution is unknown, and minimizing the risk is not possible. 


Suppose that we have some exemplary samples from the distribution. 
Specifically, consider n samples X;, Y;;"_, distributed independently and 
identically (iid) according to the otherwise unknown Px.y. Let us call these 
samples training data, and denote the collection by D, = X;, Y;;_,. Let's 
also define a collection of candidate mappings .Y. We will use the training 
data D,, to pick a mapping f,, € ¥Y that we hope will be a good predictor. 
This is sometimes called the Model Selection problem. Note that the 
selected model f,, is a function of the training data: 

Equation: 


fn (X) = f (X; Dn), 


which is what the subscript n in f,, refers to. The risk of f,, is given by 
Equation: 


R(fn) = Exy [€(fn(X), Y)I- 


Note that since f,, depends on D,, in addition to a new random pair (X,Y), 
the risk is a random variable (i.e., a function of the training data D,,). 
Therefore, we are interested in the expected risk, computed over random 
realizations of the training data: 

Equation: 


Ep, |R(fn)I- 


We hope that f,, produces a small expected risk. 


The notion of expected risk can be interpreted as follows. We would like to 
define an algorithm (a model selection process) that performs well on 
average, over any random sample of n training data. The expected risk is a 
measure of the expected performance of the algorithm with respect to the 
chosen loss function. That is, we are not gauging the risk of a particular 
map f € F, but rather we are measuring the performance of the algorithm 
that takes any realization of training data and selects an appropriate model 


eee 


This course is concerned with determining “good” model spaces ¥ and 
useful and effective model selection algorithms. 


Introduction to Classification and Regression 


Pattern Classification 


Recall that the goal of classification is to learn a mapping from the feature space, 2’, to a label space, Y. This 
mapping, f, is called a classifier. For example, we might have 
Equation: 


® = R! 
Y = {0,1}. 


We can measure the loss of our classifier using 0 — 1 loss; i.e., 
Equation: 


. if 
£89) = 154) = . 


<> & 
HK 
e ec 


Recalling that risk is defined to be the expected value of the loss function, we have 
Equation: 


R(f) = Exy[€(f(X), Y)] = Exy [lgxyev}] = Pxy(f(X) #Y). 


The performance of a given classifier can be evaluated in terms of how close its risk is to the Bayes' risk. 
(Bayes' Risk) 

The Bayes' risk is the infimum of the risk for all classifiers: 

Equation: 


R’ =inf R(f). 


We can prove that the Bayes risk is achieved by the Bayes classifier. 


Bayes Classifier 
The Bayes classifier is the following mapping: 


Equation: 
* 1, n(x) > 1/2 
Peafh Wz 
0, otherwise 
where 
Equation: 


n(x) = Pyx(Y =1|X =2). 
Note that for any x, f” (a) is the value of y € {0,1} that maximizes Pyy (Y = y|X = 2). 
Theorem 


Risk of the Bayes Classifier 
Equation: 


Let g(x) be any classifier. We will show that 
Equation: 


P(g(X) #Y|X =2) >P(f (x) #Y|X=2). 


For any g, 
Equation: 
P(g(X) #Y|X=2) = 1-P(¥ =9(X)|X =2) 
2 111k Saye’ S00 Sax =o) 
= 1-[B [Leal gon=u|X = 2] + B [14x=0}1 to(x)-05|% = @]] 
[1 fo(2)=1) 8 [Dyy—ay|X = a] + 1 g(x) F [1r—o|X = a] 
= 1- bivneraed = 1|x = xr) + 1 ¢(z)=03P(Y = O|.X = x) 
[1 fo(2)=1)7 (2) + 1 gg(x)=0}(1 — n(2))] 


Next consider the difference 
Equation: 


P(g(2) # ¥|X = 2) — P(f" (2) AY|X ==) 


n(x) [dcr ce)-a} 3 1 g2)=1] + (1—n(2)) 12 ¢4°¢2)-0} 7 
= (2) [Ley (ayaay — Vera] — 0-0 (@)) [1 eyeeyany - 
(2n(x) — 1) Agra} = Ligz)=1)); 


where the second equality follows by noting that 1 ¢4(2)-o} = 1 — 1¢g(z)=1;. Next recall 
Equation: 


: at n(x) > 1/2 


0, otherwise 


For x such that n(x) > 1/2, we have 
Equation: 


(2n(x) — (2019 — Lig(e)=1} 
i ee-—1 —~-— orl 
= CO OTN 


>0 


and for x such that (a) < 1/2, we have 
Equation: 


(2n(x) — 1) (sr. SAG Ga ) 
——~ _—~—" 0or1 


— 
<0 ¥ 0 
<0 


which implies 
Equation: 


(2n(z) — 1) uw} = 1 ae)=1) 20 


or 
Equation: 


P(g(X) #Y|X =2) SP le (2) #Y|X =<). 


Note that while the Bayes classifier achieves the Bayes risk, in practice this classifier is not realizable because we 
do not know the distribution Pxy and so cannot construct (2). 


Regression 
The goal of regression is to learn a mapping from the input space, 2, to the output space, &. This mapping, f, is 
called a estimator. For example, we might have 
Equation: 
gS Re 
Y= R. 


We can measure the loss of our estimator using squared error loss; i.e., 
Equation: 


£(i,y) = (y-9)’- 


Recalling that risk is defined to be the expected value of the loss function, we have 
Equation: 


R(f) = Exy [e(f(X),¥)] = Exy [(f(X) - ¥)’]. 


The performance of a given estimator can be evaluated in terms of how close the risk is to the infimum of the risk 
for all estimator under consideration: 
Equation: 


R’ =inf R(f). 


Theorem 
Minimum Risk under Squared Error Loss (MSE) 


Let f* (2) = Byjx [¥|X =a] 
Equation: 


Equation: 


Rf) = Exy[(f(X)-¥)| 
= Bx[Byx|(f(%) - YX] 
= Bx[Byix|(f(X) — Byx [YX] + Brix [VIX] - ¥)"1]] 
Ex( Byix[(f(X) - Eryx ¥1X))"|X] 


= +2Ey\x | (f(X) — Eyjx [Y|X]) (Eyyx [YX] — Y)|X] 
nee |(Evix IY |X] — y)’|x]] 


Ex{ Ey.x | (F(X) - Byx[¥1X1)” |X] 
= -+$2(f (X) - Evix [Y1X)) x 0 
+Eyjx | (Eyix YX] - Y)’|X]] 


= Exy|(f(X) - Byx Y1x1)"] +R (s’). 


Example: 
Thus if f (x) = Ey\x [Y|X = a], then R (a) — R’, as desired. 


Empirical Risk Minimization 


Empirical Risk 
Let {X;, Yi} be Pxy bea collection of training data. Then the empirical risk is defined as 
Equation: 


Raf) = = S2e(F(4),¥0. 


Empirical risk minimization is the process of choosing a learning rule which minimizes the empirical risk; 


Le., 
Equation: 
f, =argmin R, (f). 
fn =argmin Rn (f) 
Example: 


Pattern Classification 
Let the set of possible classifiers be 
Equation: 


F = {e+ sign (w'z) : we R*} 


and let the feature space, 2, be [0, 1]* or R“. If we use the notation f, () = sign (w'z), then the set of 
classifiers can be alternatively represented as 
Equation: 


a Afpeane Rey. 


In this case, the classifier which minimizes the empirical risk is 
Equation: 


fn = rer Rn (f) 


I 


iS 
argmin, — yy 1 {sign(w'X,) AY} 


Example linear classifier 
for two-class problem. 


Example: 

Regression 

Let the feature space be 
Equation: 


X =(0,1| 


and let the set of possible estimators be 
Equation: 
F = {degree d polynomials on (0, 1}. 
In this case, the classifier which minimizes the empirical risk is 
Equation: 


n~ 


fn 


I 


angmuin Fn (7) 


argmin es sy CAC a 


feF n =a 


I 


Alternatively, this can be expressed as 
Equation: 


§) 
I 


d41 
weR na 


= arg min || Vw—Y ||? 
weRu1 


where V is the Vandermonde matrix 


Equation: 
1 pe 
1 Xs xe 
VS ; 
1 xX; x 


The pseudoinverse can be used to solve for @ : 
Equation: 


@ = (V'V) VY. 


A polynomial estimate is displayed in [link]. 
k=38 


14 T T T 7 T T T 


Example polynomial estimator. Blue curve denotes 
if sr magenta curve is the polynomial fit to the data 
(denoted by dots). 


Overfitting 


n 
arg min 2 en + wyX;+... 


Suppose F, our collection of candidate functions, is very large. We can always make 


Equation: 


min R, (f) 


smaller by increasing the cardinality of Y, thereby providing more possibilities to fit to the data. 


Consider this extreme example: Let ¥ be all measurable functions. Then every function f for which 
Equation: 


Y; x= X; fori=l,...,n 
f(x)= , 
any value, otherwise 


has zero empirical risk (R, (f) = 0). However, clearly this could be a very poor predictor of Y for a new input X 


Example: 
Classification Overfitting 
Consider the classifier in [link]; this demonstrates overfitting in classification. If the data were in fact generated 
from two Gaussian distributions centered in the upper left and lower right quadrants of the feature space domain, 
then the optimal estimator would be the linear estimator in [link]; the overfitting would result in a higher 
robability of error for predicting classes of future observations. 
. 


Example of overfitting 
classifier. The classifier's 
decision boundary wiggles 
around in order to 
correctly label the training 
data, but the optimal 
Bayes classifier is a 
straight line. 


Example: 

Regression Overfitting 

Below is an m-file that simulates the polynomial fitting. Feel free to play around with it to get an idea of the 
overfitting problem. 


% poly fitting 
% rob nowak 1/24/04 


clear 
close all 


% generate and plot "true" function 

t = (0:.001:1)'; 

f = exp(-5*(t-.3).42)+.5*exp(-100*(t-.5).42)+.5*exp(-100*(t-.75).42); 
figure(1) 

plot(t,f) 


% generate n training data & plot 
n = 10; 
sig = 0.1; % std of noise 


xX = .97*rand(n,1)+.01; 

y = exp(-5*(xX-.3).42)+.5*exp(-100*(x-.5).42)+.5*exp(-100* 
(x-.75).42)+sig*randn(size(x)); 

figure(1) 

clf 

plot(t,f) 

hold on 


plot(x,y,'.') 


% fit with polynomial of order k (poly degree up to k-1) 
k=3; 
for i=1:k 
V(:,1) = xX.A4(1-1); 
end 
p = inv(V'*V)*V'*y; 


for i=1:k 
Vt(:,1) = t.A(i-1); 
end 
yh = Vt*p; 
figure(1) 
clf 
plot(t,f) 
hold on 
plot(x,y,'.') 
plot(t,yh, 'm' ) 


Example polynomial fitting problem. Blue curve is f a magenta curve is the polynomial fit to the data (dots). 
(a) Fitting a polynomial of degree d = 0: This is an example of underfitting (b)d = 2 (c)d = 4(d)d = 6: 
This is an example of overfitting. The empirical loss is zero, but clearly the estimator would not do a good 

job of predicting y when z is close to one. 
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Introduction to Complexity Regularization 


Competing Goals: The Bias- Variance Tradeoff 


We ended the previous lecture with a brief discussion of overfitting. Recall that, 
given a set of n data points, D,,, and a space of functions (or models) #, our goal 


in solving the learning from data problem is to choose a function fr € F which 
minimizes the expected risk & R (Fn) , where the expectation is being taken 


over the distribution Pxy on the data points D,. One approach to avoiding 
overfitting is to restrict Y to some subset of all measurable function. To gauge the 
performance of a given f in this case, we examine the difference between the 
expected risk of f and the Bayes' risk (called the excess risk). 

Equation: 


E|R(f.)|-R" = (£[R(fa)|—infrex R(f)) + (infyes R(f) — RB’) 


estimation error approximation error 


The approximation error term quantifies the performance hit incurred by 
imposing restrictions on .Y. The estimation error term is due to the randomness 


of the training data, and it expresses how well the chosen function fr will perform 
in relation to the best possible f in the class Y. This decomposition into stochastic 
and approximation errors is similar to the bias-variance tradeoff which arises in 
classical estimation theory. The approximation error is like a bias squared term, 
and the estimation error is like a variance term. By allowing the space Ato be 
large[ footnote] we can make the approximation error as small as we want at the 
cost of incurring a large estimation error. On the other hand, if is very small 
then the approximation error will be large, but the estimation error may be very 
small. This tradeoff is illustrated in [link]. 

When we say is large, we mean that |.¥|, the number of elements in F, is large. 


estimation 
error 


approximation 
error 


Complexity of F 


Illustration of tradeoff between estimation and 
approximation errors as a function of the size 
(complexity) of the ¥. 


Why is this the case? We do not know the true distribution Pxy on the data, so 
instead of minimizing the expected risk of we design a predictor by minimizing 
the empirical risk: 


Equation: 
Fn _ argmin Rn (f); 
a ee 
Ra(f) = = dS (F (Xi), ¥i). 
i=1 


If Fis very large then R,, (f) can be made arbitrarily small and the resulting f,, 


can “overfit” to the data since R,, (f) is not a good estimator of the true risk 


R(fn). 


Prediction 4) 
Error \ 


«< lrue risk 


ampirical risk 


oo ~ 


= > 
underfitting overtitting Complexity 


Best 
Madel 


Illustration of empirical risk and the problem of 
overfitting to the data. 


The behavior of the true and empirical risks, as a function of the size (or 
complexity) of the space F, is illustrated in [link]. Unfortunately, we can't easily 
determine whether we are over or underfitting just by looking at the empirical risk. 


Strategies To Avoid Overfitting 


Picking 
Equation: 


f argmin (f) 


is problematic if Ais large. We will examine two general approaches to dealing 
with this problem: 


1. Restrict the size or dimension of A(e.g., restrict Yto the set of all lines, or 
polynomials with maximum degree d). This effectively places an upper 
bound on the estimation error, but in general it also places a lower bound on 
the approximation error. 

2. Modify the empirical risk criterion to include an extra cost associated with 
each model (e.g., higher cost for more complex models): 

Equation: 


~ 


f. = argmin {R,(f) + 0(/)}. 


The cost is designed to mimic the behavior of the estimation error so that the 
model selection procedure avoids models with a estimation error. Roughly 
this can be interpreted as trying to balance the tradeoff illustrated in [link]. 
Procedures of this type are often called complexity penalization methods. 


Example: 

Revisit the polynomial regression example (Lecture 2, Ex. 4), and incorporate a 
penalty term C'(f) which is proportional to the degree of f, or the derivative of f. 
In essence, this approach penalizes for functions which are too “wiggly”, with the 
intuition being that the true function is probably smooth so a function which is 
very wiggly will overfit the data. 

How do we decide how to restrict or penalize the empirical risk minimization 
process? Approaches which have appeared in the literature include the following. 


Method of Sieves 


Perhaps the simplest approach is to try to limit the size of Ain a way that depends 
on the number of training data n. The more data we have, the more complex the 
space of models we can entertain. Let the class of candidate functions grow with n 
. That is, take 

Equation: 


F,, F2,°++,;Fny °° 


where |.F;| grows as 1 — oo. In other words, consider a sequence of spaces with 
increasing complexity or degrees of freedom depending on the number of training 
data samples, n. 


Given samples {X;, Y;};__, i-i.d. distributed according to Pxy, select f € F,, to 
minimize the empirical risk 
Equation: 


f, = in R, (f). 
f argmin (f) 


In the next lecture we will consider an example using the method of sieves. The 
basic idea is to design the sequence of model spaces in such a way that the excess 
risk decays to zero as n —> oo. This sort of idea has been around for decades, but 
Grenander's method of sieves is often cited as a nice formalization of the idea: 
Abstract Inference, Wiley, New York. 


Complexity Penalization Methods 


Bayesian Methods 


In certain cases, the empirical risk happens to be a (log) likelihood function, and 
one can then interpret the cost C(f) as reflecting prior knowledge about which 
models are more or less likely. In this case, e CV) is like a prior probability 
distribution on the space ¥. The cost C(f) is large if f is highly improbable, and 
C(f) is small if f is highly probable. 


Alternatively, if we restrict Ato be small, and denote the space of all measurable 
functions as F = ¥ U F¥*, then it is essentially as if we have placed a uniform 
prior over all functions in ¥Y, and zero prior probability on the functions in ¥°. 


Description Length Methods 


Description length methods represent each f with a string of bits. More 
complicated functions require more bits to represent. Accordingly, we can then set 
the cost c(f) proportional to the number of bits needed to describe f (the 
description length). This results in what is known as the minimum description 
length (MDL) approach where the minimum description length is given by 
Equation: 


min {Rn (f) + O(/)}. 


In the Bayesian setting, p(f) « e~C(f) can be interpreted as a prior probability 
density on Y, with more complex models being less probable and simpler models 


being more probable. In that sense, both the Bayesian and MDL approaches have a 
similar spirit. 


Vapnik-Cervonenkis Dimension 


The Vapnik-Cervonenkis (VC) dimension measures the complexity of a class F 
relative to a random sample of n training data. For example, take “to be all linear 
classifiers in 2-dimensional feature space. Clearly, the space of linear classifiers is 
infinite (there are an infinite number of lines which can be drawn in the plane). 
However, many of these linear classifiers would assign the same labels to the 
training data. 


The number of unique labellings of the training data that can be achieved with 
linear classifiers is, in fact, finite. A line can be defined by picking any pair of 
training points, as illustrated in [link]. Two classifiers can be defined from each 
such line: one that outputs a label “1” for everything on or above the line, and 
another that outputs “O” for everything on or above. There exist ( such pairs of 
training points, and these define all possible unique labellings of the training data. 
Therefore, there are at most 2 (3) unique linear classifiers for any random set of n 
2-dimensional features (the factor of 2 is due to the fact that for each linear 
classifier there are 2 possible assignments of the labelling). 


Fitting a linear 
classifier to 2- 
dimensional data. 
There are an infinite 
number of such 


classifiers. We can 
generate a linear 
classifier by choosing 
two data points, 
drawing a line with 
both points on one 
side, and declaring all 
points on or above 
the line to be “+1” 
(or “—1”) and all 
points below the line 
to be “—1” (or “+1 


»), 


From the discussion 
in the previous figure, 
we see that the two 
linear classifiers 
depicted in this figure 
are equivalent for this 
set of data points, and 
hence relative to the 
set of n training data 
there are only on the 
order of n? unique 
linear classifiers. 


Thus, instead of infinitely many linear classifiers, we realize that as far as a 
random sample of n training data is concerned, there are at most 


Equation: 
9 & = 2n! 
2/  (n— 2)!2! 


= n(n-1) 


unique linear classifiers. That is, using linear classification rules, there are at most 
n(n—1)® n? unique label assignments for n data points. If we like, we can 
encode each possibility with log, n(n — 1) © 2 log, n bits. In d dimensions 
there are 2) hyperplane classification rules which can be encoded in roughly 

d log, n bits. Roughly speaking, the number of bits required for encoding each 
model is the VC dimension. The remarkable aspect of the VC dimension is that it 
is often finite even when -F is infinite (as in this example). 


If &has d dimensions in total, we might consider linear classifiers based on 
1,2,---,d features at a time. Lower dimensional hyperplanes are less complex 
than higher dimensional ones. Suppose we set 

Equation: 


| 


Fy 


YF = linear classifiers using 2 features. 


linear classifiers using 1 feature 


and so on 


These spaces have increasing VC dimensions, and we can try to balance the 
empirical risk and a cost function depending on the VC dimension. Such 
procedures are often referred to as Structural Risk Minimization. This gives you 
a glimpse of what the VC dimension is all about. In future lectures we will revisit 
this topic in greater detail. 


Hold-out Methods 


The basic idea of “hold-out” methods is to split the n samples D = {.X;, Yi}"_, 
into a training set, Dr, and a test set, Dy. 
Equation: 


DRANK GG) a LDV) Cae 


Now, suppose we have a collection of different model spaces {.F } indexed by 
A € A(e.g., F) is the set of polynomials of degree d, with \ = d), or suppose 
that we have a collection of complexity penalization criteria LD (f) indexed by A 
(e.g.,let Ly (f) = R(f) + Ac(f), with A € R*). We can obtain candidate 
solutions using the training set as follows. Define 


Equation: 
Ratt = > 2G a) 
i=1 
and take 
Equation: 
fy = argmin R,, (f) 
CF) 

or 

Equation: 


—~ 


fy = argmin {Ry (f) +Arc(A)}. 


This provides us with a set of candidate solutions th \. Then we can define the 


hold-out error estimate using the test set: 
Equation: 


1 n 
ee aay | De £(f (Xi), Yi), 


1=m+1 


Ry(f) = 


and select the “best” model to be f = f; where 
Equation: 


—~ 


A= argmin Ry (fi). 


This type of procedure has many nice theoretical guarantees, provided both the 
training and test set grow with n. 


Leaving-one-out Cross-Validation 


A very popular hold-out method is the so call “leaving-one-out cross-validation” 
studied in depth by Grace Wahba (UW-Madison, Statistics). For each A we 


compute 
Equation: 
fe = argmin + S7(f(Xi),¥i) +40 (f) 
A fer 
ifk 
or 
Equation: 


fy im Sel P(X). Y; 
ty argmin “pe (f (X;), Yi). 
itk 


Then we have cross-validation function 
Equation: 


va) = 2>re(#° (%),¥) 


* 


A = argmin V (A). 


Summary 


To summarize, this lecture gave a brief and incomplete survey of different 
methods for dealing with the issues of overfitting and model selection. Given a set 
of training data, D,, = {X;, Y;}\_,, our overall goal is to find 

Equation: 


i =argmin R (f) 


from some collection of functions, .. Because we do not know the true 
distribution Pyy underlying the data points Dy, it is difficult to get an exact 
handle on the risk, R(f). If we only focus on minimizing the empirical risk R (f) 
we end up overfitting to the training data. Two general approaches were presented. 


= 


. In the first approach we consider an indexed collection of spaces {.F} <4 


such that the complexity of Y increases as increases, and 
Equation: 


lim ¥) = F¥. 
A> 00 


A solution is given by 
Equation: 


Fy =arg min R,, A) 


fEF\* 


where either is a function which increases with n, 
Equation: 


or A” is chosen by hold-out validation. 


. The alternative approach is to incorporate a penalty term into the risk 


minimization problem formulation. Here we consider an indexed collection 
of penalties {C')},_, satisfying the following properties: 


1 Cy : Fo R'; 
2. For each f € ¥ and ry < Az we have C), (f) < C), (f); 
3. There exists Ag € A such that Cy, (f) = 0 forall f € F. 


In this formulation we find a solution 
Equation: 


fy = argmin Rn (f) + Cy (f); 


where either \° = » (n), a function growing the number of data samples n, 
or A” is selected by hold-out validation. 


Consistency 


If an estimator or classifier f* satisfies 
Equation: 


E|R(f.)| ee) as N —> 00, 


then we say that f)« is A-consistent with respect to the risk R. When the context 
is clear, we will simply say that f is consistent. 


An Example of the Use of Sieves for Complexity Regularization in Denoising 


Consider the following setting. Let 
Equation: 


Y=f (X)+W, 
where X is a random variable (r.v.) on 2 = [0,1], W isar.v. on Y = R, independent 


of X and satisfying 
Equation: 


E([W]=0 and E[W?] =0* <oo. 
Finally let f° : [0,1]  Rbea function satisfying 
Equation: 


f(t) — f* (s)| < Lit — |, Vt,s ¢ [0,1], 


where L > 0 is aconstant. A function satisfying condition [link] is said to be Lipschitz 
on |0, 1]. Notice that such a function must be continuous, but it is not necessarily 
differentiable. An example of such a function is depicted in [link](a). 


Example of a Lipschitz function, and our observations setting. (a) random 
sampling of f *\ the points correspond to (X;, Y;), 7 =1,..., 7; (b) deterministic 
sampling of f”, the points correspond to (i/n, Y;), 7 =1,...,n. 


Note that 
Equation: 


E\Y|X =a 


E| f* (X)+W|X=2| 


El f* (2) +W|x =a 
f(z) + E[W] =f (2). 


Consider our usual setup: Estimate f* using n training examples 


Equation: 
n iid. 
ene Gy een ~ Pxy, 
Y; = f (Xi) + Wi, i= {1,...,n}, 


where “* means independently and identically distributed. [link](a) illustrates this 


setup. 


In many applications we can sample 2 = [0,1] as we like, and not necessarily at 
random. For example we can take n samples uniformly on [0,1] 
Equation: 


ot = Leg Ths 


1 
Li — 
n 


f (ai) + Wi 
(=) +W;. 
n 


We will proceed with this setup (as in [link](b)) in the rest of the lecture. 


on 
| 


Our goal is to find f,, such that E|| 7 =. | — 0, as n — 0 (here || - || is the 
: ~ (2 
f ()-fr(d)| ae), 


usual L2-norm; ice., || f° — em \|?= Us 


Let 
Equation: 


F={f: f is Lipschitz with constant L}. 


The Risk is defined as 
Equation: 


f(t) — F(a. 


R= F'-4 1? = f 


The Expected Risk (recall that our estimator fr is based on {x;, Y;} and hence is a r.v.) 
is defined as 
Equation: 


BR (fa)| = ll f° — fe I?). 


Finally the Empirical Risk is defined as 
Equation: 


Let 0 < m, < m2 < m3 < -:- bea sequence of integers satisfying m,, —> 00 as 

n — oo, andk,m, = n for some integer k,, > 0. That is, for each value of 7n there is 
an associated integer value m,. Define the Sieve ¥1, Fo, F3, 
Equation: 


ceey 


Fy = We f(t) = alias: oj € Rt 


j=l 


Fy, is the space of functions that are constant on intervals 
Equation: 


From here on we will use m and k instead of m,, and k,, (dropping the subscript n) for 
notational ease. Define 
Equation: 


my 4 1 4 
fr (t) = do Ttter,,}, Where c;= Z De f (=). 
Note that f, € Fry. 


Example: 
Exercise 1 
Upper bound || f° — fy, ||’. 
Equation: 


ag nO 1 i: - 2 
ifr = ff o-t0/ee 
ENDS * (t) — fa (t)| at 
vf re fn (6) 
ay "(t) —c,|'dt 
yee 
2 
= a 1 * 1 
= eh f Tee (=) dt 
2 
= 1 : me 
= el ilo ®-#(=)) dt 
2 
€ 1 : en 
2S) le S FPe-Pelile 
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The above implies that || f" — fy ||? 0 as n — 00, since 


m= Mn — co as n — oo. In words, with n sufficiently large we can approximate f° 
to arbitrary accuracy using models in F,, (even if the functions we are using to 
approximate f * are not Lipschitz!). 


For any f € Fn, f = 05-1 Cj Ltter,,,}» We have 
Equation: 


Let fin =argminy-g, Rp (f). Then 


Equation: 
f(t) =) [@;Uftctn} Where @} = ; » ¥ 
j=l :+Eljm 
Example: 
Exercise 2 
Show [link]. 


Note that E [é;] = c, and therefore E Fn (t)| = f, (t). Lets analyze now the 


expected risk of fa: 
Equation: 


E|Il f° - fa 1? | 


Ell f° - fat fn - fa IP 
= Wf" = fo ll? +B fo — fa II] +28 [(F" — fos Fu — Fr) 
I f° - fo |? +2 fa — fa lI?) +2 f° - tus B [fn — fr) 
= If - fall? +2lll fo - fa I"); 


where the final step follows from the fact that F Fn (t)| = f, (t). A couple of 


important remarks pertaining the right-hand-side of equation [link]: The first term, 

|| f- — fn ||? corresponds to the approximation error, and indicates how well can we 
approximate the function f~ with a function from F,. Clearly, the larger the class Fy, 
is, the smallest we can make this term. This term is precisely the squared bias of the 
estimator f,,. The second term, £7 | fn — Fn | , is the estimation error, the variance 
of our estimator. We will see that the estimation error is small if the class of possible 
estimators ¥,, is also small. 


The behavior of the first term in [link] was already studied. Consider the other term: 
Equation: 


E||| fo — fo IP] 
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Combining all the facts derived we have 
Equation: 
eer L* m 1 om 
ae ee ee O(max {5,2}, 
m n m2 n 


This equation used Big-O notation. 

What is the best choice of m? If m is small then the approximation error (i.e., 

O(1 ye m?)) is going to be large, but the estimation error (i.e., O(m/n)) is going to be 
small, and vice-versa. This two conflicting goals provide a tradeoff that directs our 
choice of m (as a function of 7). In [link] we depict this tradeoff. In [link](a) we 
considered a large m,, value, and we see that the approximation of f by a function in 
the class ¥,, can be very accurate (that is, our estimate will have a small bias), but 
when we use the measured data our estimate looks very bad (high variance). On the 
other hand, as illustrated in [link](b), using a very small m,, allows our estimator to get 
very close to the best approximating function in the class ¥,,, so we have a low 
variance estimator, but the bias of our estimator (i.e., the difference between f,, and f-) 
is quite considerable. 


Approximation and estimation of f” (in blue) for n = 60. The function f,, is 


depicted in green and the function fa is depicted in red. In (a)we have m = 60 
and in (b) we have m = 6. 


We need to balance the two terms in the right-hand-side of [link] in order to maximize 
the rate of decay (with n) of the expected risk. This implies that aa = > therefore 


my, = n'/3 and the Mean Squared Error (MSE) is 
Equation: 


Ell fa — fa \I?] = O(n). 


So the sieve ¥1, Fo, --- with 
Equation: 


A sie {f10= S ata acep ser}, 
j=l My =" ~ Mn 


produces a ¥-consistent estimator for f° = E[Y|X +a] € F. 

It is interesting to note that the rate of decay of the MSE we obtain with this strategy 
cannot be further improved by using more sophisticated estimation techniques (that is, 
n 2/3 is the minimax MSE rate for this problem). Also, rather surprisingly, we are 
considering classes of models ¥,, that are actually not Lipschitz, therefore our 
estimator of f "is nota Lipschitz function, unlike f itself. 


Plug-In Classifier and Histogram Classifier 


We return to the topic of classification, and we assume an input (feature) space 2 and a binary output (label) 
space Y = {0, 1}. Recall that the Bayes classifier (which minimizes the probability of misclassification) is 
defined by 
Equation: 
* 1, PY =X =2)>1/2 
Ces i ae 


0, otherwise 


Throughout this section, we will denote the conditional probability function by 
Equation: 


n(x) = P(Y =1|X =2). 


Plug-in Classifiers 


One way to construct a classifier using the training data {X;, Y;}",_, is to estimate (x) and then plug-it into 


the form of the Bayes classifier. That is obtain an estimate, 
Equation: 


fin (2) = 0 (23 {Xi Vida) 


and then form the “plug-in" classification rule 
Equation: 


fay-{h e217 


0, otherwise © 


Note: The function 7(z) is generally more complicated than the ultimate classification rule (binary-valued), 
as we can see 


Equation: 


n : &-(0,1] 
f : {0,1} 


Therefore, in this sense plug-in methods are solving a more complicated problem than necessary. However, 
plug-in methods can perform well, as demonstrated by the next result. 

Theorem 

Plug-in Classifier 


Let 7 be an approximation to 7, and consider the plug-in rule 
Equation: 


re {* a(x) > 1/2 


0, otherwise 


Then, 
Equation: 
R(f)— RB’ < 2E[|ln(z) — A(z) |] 


where 
Equation: 


a 
i 
a) 

I 


P(f(X) #Y) 


R* = R(f ) =inf R(f) 
Consider any x € R?. In proving the optimality of the Bayes classifier f * in Lecture 2, we showed that 
Equation: 


P(f(2) #Y|X = 2) — P(f* (2) AYIX=2) = (2nle)-1[Igpeyay —Lyw=y); 


which is equivalent to 
Equation: 
P(f(z) #¥|X =2) - P(f* (2) AYIX=2) = 2nl2)-Uley~epayys 


since f° (x) = 1 whenever 2n(x) — 1 > 0. Thus, 
Equation: 


PURX)AY)- R= ff 2in(@) 1/211 weneyPx (ade 


where px (x) is the marginal density of X 


IA 


[2n@ AON yrweneyex (de 


[20 @) A @)ipx (2)ae 
= 28lIn(X) — 4(X)]] 


IA 


where the first inequality follows from the fact 
Equation: 


f(x) AF (x) => |n(x)-A(2)| = In(w) — 1/2| 


and the second inequality is simply a result of the fact that 1 {f° ()#F(a)} is either 0 or 1. 


NX) Ae 
1/2 


0 X 


Pictorial illustration of |7 (x)—7 (x)| > |n (x) — 1/2| when 
f(x) Af (a). Note that the inequality 
P(f(X) £Y)— RS fas2ln(e) —A(@)LepayepayyPx (w)de 
shows that the excess risk is at most twice the integral over the set 
where f° (x) # f (x). The difference |7 (x)—7(a)| may be 
arbitrarily large away from this set without effecting the error rate of 
the classifier. This illustrates the fact that estimating 7 well 
everywhere (i.e., regression) is unnecessary for the design of a good 
classifier (we only need to determine where 7 crosses the 1/2-level). 
In other words, “classification is easier than regression.” 


The theorem shows us that a good estimate of 7 can produce a good plug-in classification rule. By “good" 
estimate, we mean an estimator # that is close to 7 in expected Lj-norm. 


The Histogram Classifier 


Let's assume that the (input) features are randomly distributed over the unit hypercube 2 = (0, 1]? (note that 
by scaling and shifting any set of bounded features we can satisfy this assumption), and assume that the 
(output) labels are binary, i.e., Y = {0,1}. A histogram classifier is based on a partition the hypercube (0, i" 
into M smaller cubes of equal size. 


Example: 

Partition of hypercube in 2 dimensions 

Consider the unit square {0, 1] ? and partition it into MW subsquares of equal area (assuming M is a squared 
integer). Let the subsquares be denoted by {Q;}, i = 1,..., M. 


=1/2 


0 ’ l 
m2 


Example of hypercube (0, Ne in M equally 
sized partition 


Define the following piecewise-constant estimator of (zx): 


Equation: 
M =~ 
fin (x) = SS Pil treq;} 
j=l 
where 
Equation: 


es YS 1{x,€Q;,¥:=1} 
P; = m4 ; 
ey {X;€Q;} 


Like our previous denoising examples, we expect that the bias of 77, will decrease as M increases, but the 
variance will increase as / increases. 


Theorem 
Consistency of Histogram Classifiers 


If M — oo and 47 — 00 as n — oo, then the histogram classifier risk converges to the Bayes risk for every 
distribution Pxy with marginal density px (x) > c, for some constant c > 0.[footnote]. 

Actually, the result holds for every distribution Pxy. For the more general theorem, refer to Theorem 6.1 in A 
probabilistic Theory of Pattern Recognition by Luc Devroye, Laszl6 Gy6rfi and Gabor Lugosi. 


What the theorem tells us is that we need the number of partition cells to tend to infinity (to insure that the bias 
tends to zero), but they can't grow faster than the number of samples (i.e., we want the number of samples per 
box tending to infinity to drive the variance to zero). 


r ax)dx a 
Soyelpx(e)de (the theoretical analog of P;) and define 


met oz = So, px(x)dx 


Equation: 


M 
n(x) = + P51 {2<Q;} 
j=l 


The function 7 is the theoretical analog of 7 (i.e., the function obtained by averaging 7 over the partition cells). 
By the triangle inequality, 
Equation: 


El \fim(X)—n(X)|] < Ellfin(X) —7(X)|] + Ellin X) — 0 (4) 


EstimationError ApproximationError 


Let's first bound the estimation error. For any x € [0, 1]*, let Q(«) denote the histogram bin in which z falls 
in. Define the random variable 
Equation: 


N (2) => 1¢x,<a@)} 
t=1 


If Q («) = Q,, then this random variable is simply nP;. Note that 
Equation: 


where B(x) == 77-4 1px,cq(e),¥i=1} = 2i-x,<Q(x) Yi- B(@) is simply the number of samples in cell Q(z) 
labelled 1. Now 7, (x) is a fairly complicated random variable, but the conditional distribution of B(x) given 
N(z) is relatively simple. Note that 

Equation: 


B(a)| N(x) =k ~ Binomial(k, 7 (x)) 
since 7 (x) is the probability of a sample in Q(z) having the label 1 and we are conditioning on the event of 
observing & samples in Q(z). 


Now consider the conditional expectation 
Equation: 


E| Fe _ 7 (2)| | N (2) =, k>0 
E || (x) — 77 (a)| | N (2) =k] < 


1, k=0 (since 0 <7(zx) <1) 


Next note that 
Equation: 


=||70 -a(e)| | N(@)=#] = | =) — a(a)|| we) = 4 
= B|+/B(2) — a(2) voy 
E|B(z)] 


IA 


2 [a — kii(x)|” | N (2) = j 


conditional variance of B(x) 


by the Jensen's inequality, E [|Z|] < (E[|Z|]) 7, 


Therefore, 
Equation: 
Bl Fey 7] NC) = 8] < Z(en(@) (2 n(2)))* 
(x) A= 7 (2) 
sq 
and 
Equation: 


B[|fn (2) — 7 (2)| | N(z) =] < | 


or in other words, 
Equation: 


E|\fin (x) — 71 («)||N (2) =k] < 


7 (x) (1—7 (x) 
= NG). Lpn(2)>0} + 1n(e)=0} 


Now taking expectation with respect to N(x) 
Equation: 


En[2[lin (2) -9(2)|N (0) =A] < es| EEF en + P(N (2) =0) 


N(z) 
1 
< 2 | ton + P(N (x) =0) 
< = P(N (2) <8) + —_P(N(2) >) + P(N (2) =0) 


Now a key fact is that for any k > 0, P(N < k) — 0 as n — oo. This follows from the assumption that the 
marginal density px (x) > c, for some constant c > 0, and qr 7 coasn — ov. This result is easily verified 
by contradiction. If P(N < k) + q > 0 as n > ov, then Px (x) > 0 is contradicted. Thus, for any « > 0 
there exists a k > 0 such that Ta <eand P(N <k) < « for n sufficiently large. Therefore, for n 
sufficiently large and every x € (0, i 

Equation: 


E||fin (2) — 7 (x)|] < 3¢ 
where the expectation is with respect to the distribution of the sample {.X;, Y;};"_,. Thus, 
Equation: 


E||fm (X) —7(X)|] < 3e 


where the expectation is now with respect to the distribution of the sample and the marginal distribution of X. 


Next consider the approximation error E[|7, (X) — 7(X)|], where the expectation is over X alone. The 
function 7 may not itself be continuous, but there is another function 7, that is uniformly continuous and such 
that E||n. (X) — n(X)|] < e. Recall that uniformly continuous functions can be well approximated by 
piecewise constant functions. 


By the triangle inequality, 
Equation: 


E\\7 — nl] < Elly — mel] + Elltte — mel] + Ellme = ll 
<e <e by design 


where 7, (2) = yt a Ne (x')px (z')da'| 12<Q,}- 
Equation: 


El\n (X) — me (X)I] 


Ms 


b, In (x) — Ne (x) |px (x)dx 1 {2<Q;} 


g=1 
< € 
and since 7, is uniformly continuous, 
Equation: 
M 
Bllne(X) —ne (XI = SO i ie (2) — ne ()|Leca,ypx (2)de 
j= ij 


IA 


M 
S SP (a €Q;), where 6 depends on M 
j=l 


M 
= 6, since S>P(X€Q;)=1 
j=l 


By taking M sufficiently large, 6 can be made arbitrarily small. So for large M, 6 < e. 


Thus, we have shown 
Equation: 


E\\7n (X) — n(X)|] < 3¢ 


for sufficiently large MW. Since e > 0 was arbitrary, we have shown that taking 


Equation: 

1, fn (x) 21/2 

fr(z) = 

0, otherwise 
satisfies 
Equation: 

P(fr(X)#Y)-P(F(X) AY) < 2B [lin (X)—n(X)] 30 

if 
Equation: 


Note: P Ge (Xx) # Y) =F ft { 7x) y| is the expected risk of f, with expectation over the distributions 
of (X,Y) and {X;, Yi}"_,. 


Probably Approximately Correct (PAC) Learning 
Introduction 


Overview of the Learning Problem 


The fundamental problem in learning from data is proper Model Selection. As we have 
seen in the previous lectures, a model that is too complex could overfit the training data 
(causing an estimation error) and a model that is too simple could be a bad 
approximation of the function that we are trying to estimate (causing an approximation 
error). The estimation error arises because of the fact that we do not know the true joint 
distribution of data in the input and output space, and therefore we minimize the 
empirical risk (which, for each candidate model, is a random number depending on the 
data) and estimate the average risk again from the limited number of training samples we 
have. The approximation error measures how well the functions in the chosen model 
space can approximate the underlying relationship between the output space on the input 
space, and in general improves as the “size” of our model space increases. 


Lecture Outline 


In the preceding lectures, we looked at some solutions to deal with the overfitting 
problem. The basic approach followed was the Method of Sieves, in which the 
complexity of the model space was chosen as a function of the number of training 
samples. In particular, both the denoising and classification problems we looked at 
consider estimators based on histogram partitions. The size of the partition was an 
increasing function of the number of training samples. In this lecture, we will refine our 
learning methods further introduce model selection procedures that automatically adapt 
to the distribution of the training data, rather than basing the model class solely on the 
number of samples. This sort of adaptivity will play a major role in the design of more 
effective classifiers and denoising methods. The key to designing data-adaptive model 
selection procedures is obtaining useful upper bounds on the estimation error. To this 
end, we will introduce the idea of “Probably Approximately Correct” learning methods. 


Recap: Method of Sieves 


The method of Sieves underpinned our approaches in the denoising problem and in the 
histogram classification problem. Recall that the basic idea is to define a sequence of 
model spaces YF}, Fo, ...of increasing complexity, and then given the training data 
{X;, Y;};"_, select a model according to 

Equation: 


Fn =argmin R,, (f). 


fEFn 


The choice of the model space F,, (and hence the model complexity and structure) is 
determined completely by the sample size n, and does not depend on the (empirical) 
distribution of training data. This is a major limitation of the sieve method. In a nutshell, 
the method of sieves tells us to average the data in a certain way (e.g., over a partition of 
2&) based on the sample size, independent on the sample values themselves. 


In general, learning basically comprises of two things: 


1. Averaging data to reduce variability 
2. Deciding where (or how) to average 


Sieves basically force us to deal with (2) a priori (before we analyze the training data). 
This will lead to suboptimal classifiers and estimators, in general. Indeed deciding 
where/how to average is the really interesting and fundamental aspect of learning; once 
this is decided we have effectively solved the learing problem. There are at least two 
possibilities for breaking the rigidity of the method of sieves, as we shall see in the 
following section. 


Data Adaptive Model Spaces 


Structural Risk Minimization (SRM) 


The basic idea is to select ¥,, based on the training data themselves. Let ¥1, Fy, ...be a 
sequence of model spaces of increasing sizes/complexities with 


Equation: 

lim inf R(f)=R’. 
Let 
Equation: 


PB 
Fink argmin nF) 


be a function from ;, that minimizes the empirical risk. This gives us a sequence of 


selected models f.1, fn,2, °°: Also associate with each set F, a value C;,, > 0 that 
measures the complexity or “size” of the set ;,. Typically, C;,,, is monotonically 


increasing with k (since the sets are of increasing complexity) and decreasing with n 
(since we become more confident with more training data). More precisely, suppose that 
the C’,, chosen so that 

Equation: 


(sp (4) - R(A)| > ens] <6 


SEF, 


for some small 6 > 0. Then we may conclude that with very high probability (at least 
1 — 6) the empirical risk R,, is within C,,,,, of R uniformly on the class F;,. This type of 
bound suffices to bound the estimation error (variance) of the model selection process of 


the form R(f) < Rn (f) + Cn, and SRM selects the final model by minimizing this 
bound over all functions in J k>1 Fr. The selected model is given by ae where 


Equation: 


k —aremin {Rr (Fox) + Cot} 


A typical example could be the use of VC dimension to characterize the complexity of 
the collection of model spaces i.e.,C’,,, is derived from a bound on the estimation error. 


Complexity Regularization 


Consider a very large class of candidate models ¥. To each f € ¥ assign a complexity 
value C’, (f). Assume that the complexity value is chosen so that 
Equation: 


P(g R, (f) — R(f)| > c.() <6. 


fEF 


This probability bound also implies an upper bound on the estimation error and 
complexity regularization is based on the criterion 
Equation: 


a 


jn =argmin {R, (f) + On (A). 


Complexity Regularization and SRM are very similar and equivalent in certain instances. 
A distinguishing feature of SRM and complexity reqularization techniques is that the 
complexity and structure of the model is not fixed prior to examining the data; the data 
aid in the selection of the best complexity. In fact, the key difference compared to the 
Method of Sieves is that these techniques can allow the data to play an integral role in 
deciding where and how to average the data. 


Probably Approximately Correct (PAC) learning 


Probability bounds of the forms in [link] and [link] are the foundation for SRM and 
complexity regularization techniques. The simplest of these bounds are known as PAC 
bounds in the machine learning community. 


Approximation and Estimation Errors 


In order to develop complexity regularization schemes we will need to revisit the 


estimation error / approximation error trade-off. Let f, =argminyc.g R, (f) for some 
space of models F. 
Equation: 


R (fn) — RB = R (fn) — infyex R(f) + infyeg R(f) — R’ 


: : roximation error 
estimation Error app 


The approximation error depends on how close f is close to F, and without making 
assumptions, this is unknown. The estimation error is quantifiable, and depends on the 
complexity or size of Y. The error decomposition is illustrated in [link]. The estimation 
error quantifies how much we can “trust” the empirical risk minimization process to 
select a model close to the best in a given class. 


est err. 


total err. 


inf R(f) 


fEF R* 


approx err. 


Relationship between the errors 


Probability bounds of the forms in [link] and [link] guarantee that the empirical risk is 
uniformly close to the true risk, and using [link] and [link] it is possible to show that with 


high probability the selected model a satisfies 


Equation: 
s\ Z 
R(fn)— inf R(f) < O(n,) 
or 
Equation: 


R(fa)- inf R(f) < Cn (f). 


SEF 


The PAC Learning Model 


The estimation error will be small if R(fn) is close to infy<g¢ R (f). PAC learning 


expresses this as follows. We want - to be a “probably approximately correct” (PAC) 


a 


model from ¥. Formally, we say that f,, is € accurate with confidence 1 — 4, or (€, 6)— 
PAC for short, if 
Equation: 


P(R (Fn) - inf R(f) > :) <6. 


SEF 


This says that the difference between R (Fn) and inf r<g¢ R(f) is greater than e with 
probability less than 6. Sometimes, especially in the machine learning community, PAC 
bounds are stated as, “with probability of at least 1 — 4, |R ( i) — infyeg R(f)|< ©” 


To introduce PAC bounds, let us consider a simple case. Let A consist of a finite number 
of models, and let |.7| denote that number. Furthermore, assume that minysc.g R(f) = 0 


Example: 
F= set of all histogram classifiers with M bins => |F| = 2”. 
Equation: 


ee R(f) =0=>4 a classifier in F that has a zero probability of error 
€ 


Theorem 


Assume |.F| < oo and min;eg R(f) = 0, where R(f) = P(f(X) 4 Y). Let 


fn =argmin seg R,, (f), where R,, (f) = +. yee 1g) 4%}. Then for every n and 
E> 0, 
Equation: 


P(R(fn) >e) < |Zle™ = 6. 


Since min <.g R(f) = 0, it follows that R, (Fn) = 0. In fact, there may be several 
f © F such that Ry (f) = 0. Let Y = te R, (f) = 0}. 


Equation: 
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= P| UJ R, (f) = 0} 
fCF:R(f)>e 

< SY P(Ra(f)=0) 
fcF-R(f)>e 


A 
* 
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— 
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The last inequality follows from the fact that if R(f) = P(f(X) # Y) > e«, then the 
probability that n i.i.d. samples will satisfy f(X) = Y ce less than or equal to (1 — €)”. 
Note that this is simply the probability that R, (f) = =, Lax) 4v} = 0. Finally 
apply the inequality 1 — x < e * to obtain the deciieds result. 


Note that for n sufficiently large, 6 = |.F|e~™ is arbitrarily small. To achieve a (e, 6)- 
PAC bound for a desired € > 0 and 6 > 0 we require at least n = pie 
examples. 


Corollary 


training 


Assume that |.¥| < oo and mins. g R(f) = 0. Then for every n 
Equation: 


2 [a(j)] < me 


n 


EZ) that for a non-negative random variable Z with finite mean, 
nae a A (Z > t)dt. This follows from an application of integration by parts. 


eae 


gy 
ov 
o— 
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[P(r (Fn) > t)dt 
a (oe) 
2 : P(R( hr) >e)ar+ f P(R(fn) > t)at, for any u > 0 


Bal 


u+ (Ff e dt 
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Minimizing with respect to wu produces the smallest upper bound with u = ‘esl? 


Chernoff's Bound and Hoeffding's Inequality 
Introduction 


Motivation 


In the last lecture we consider a learning problem in which the optimal function belonged to a finite class of 
functions. Specifically, for some collection of functions A with finite cardinality | F| < co, we have 
Equation: 


min R(f)=0=> f° F. 


This is almost always not the situation in the real-world learning problems. Let us suppose we have a finite 
collection of candidate functions ¥. Furthermore, we do not assume that the optimal function f°, which 
satisfies 

Equation: 


R(f’) =inf R(F) 


where the inf is taken over all measurable functions, is a member of ¥. That is, we make few, if any, 
assumptions about f . This situation is sometimes termed as Agnostic Learning. The root of the word 
agnostic literally means not known. The term agnostic learning is used to emphasize the fact that often, 
perhaps usually, we may have no prior knowledge about f”. The question then arises about how we can 
reasonably select an f € ¥ in this setting. 


The Problem 


The PAC style bounds discussed in the previous lecture, offer some help. Since we are selecting a function 


based on the empirical risk, the question is how close is R,, (f) to R(f)Vf € F. In other words, we wish 
that the empirical risk is a good indicator of the true risk for every function in ¥. If this is case, the selection 
of f that minimizes the empirical risk 

Equation: 


i, argmin (f) 


should also yield a small true risk, that is, R(fn) should be close to min sez R(f). Finally, we can thus 


state our desired situation as 
Equation: 


P( max Rn (f) - R(A)| > c) < 6, 


— 


Rna(f)-R(f)| > 6 Vi € F. 


In this lecture, we will start to develop bounds of this form. First we will focus on bounding 


for small values of € and 6. In other words, with probability at least 1 — 6, 


P(\Ra(F) - R(f)| > c) for one fixed f € F. 


Developing Initial Bounds 


To begin, let us recall the definition of empirical risk for {X;, Y;};"_, be a collection of training data. Then 


the empirical risk is defined as 
Equation: 


Note that since the training data {X;, Y;};"_, are assumed to be i.i.d. pairs, the terms in the sum are i.i.d 
random variables. 


Let 
Equation: 


The collection of losses Bis Ae, is L.i.d according to some unknown distribution (depending on the unknown 


joint distribution of (X,Y) and the loss function). The expectation of L; is 
Elé(f (X;), Yi)| = E[é(f (X), Y)| = R(f), the true risk of f. For now, let's assume that f is fixed. 
Equation: 


B[R()] =2 ozs). YN=+ ow) = RW) 


We know from the strong law of large numbers that the average (or empirical mean) Re (f) converges 


almost surely to the true mean R(f). That is, R, (f) — R(f) almost surely as n — oo. The question is 
how fast. 


Concentration of Measure Inequalities 


Concentration inequalities are upper bounds on how fast empirical means converge to their ensemble 
counterparts, in probability. The area of the shaded tail regions in Figure 1 is P (|Rn (f) — R( f)| > c) . We 


are interested in finding out how fast this probability tends to zero as n — oo. 


Distribution of R,, (f) 


At this stage, we recall Markov's Inequality. Let Z be a nonnegative random variable. 


Equation: 
/ zp (z)dz 
0 


[ zp (z)dz+ [ zp (z)dz 


= ot f zp (z)dz 
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= tP(Z>t) 
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Take 
Equation: 


—_ 


Z=|R, (f)-R(f)| and t=e 


Equation: 
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So, the probability goes to zero at a rate of at least n~!. However, it turns out that this is an extremely loose 
bound. According to the Central Limit Theorem 
Equation: 


n 2 
R, (f) = 37 +x( a.) as m—> co 
i=1 


in distribution. This suggests that for large values of n, 
Equation: 


P(|fa()- RIN IZ) ~O(e 0), 


That is, the Gaussian tail probability is tending to zero exponentially fast. 


Chernoff's Bound 
Note that for any nonnegative random variable Z andt > 0, 
Equation: 
Z t E [e**| : : 
PUL pe P (et Se) = 7—; Vs > 0 by Markov’s inequality. 
es 


Chernoff's bound is based on finding the value of s that minimizes the upper bound. If Z is a sum of 
independent random variables. For example, say 
Equation: 


Z = SU(F(%),¥) — R(N) = (Ra ()- RUN) 


then the bound becomes 
Equation: 


(3: (L; — E[L,]) > 3 < e “Ble see Fib Dl <e “Tz Ble cee D), from independence. 


i=l i=1 


Thus, the problem of finding a tight bound boils down to finding a good bound for E|s (Li BIL; Dy. Cheroff 
('52), first studied this situation for binary random variables. Then, Hoeffding ('63) derived a more general 
result for arbitrary bounded random variables. 


Hoeffding's Indequality 


Theorem 
Hoeffding's Inequality 


Let Z1, Z2,..., Zn be independent bounded random variables such that Z; € [a;, 6;| with probability 1. Let 
Sn = S>;_, Zi. Then for any t > 0, we have 
Equation: 


at? 


P(|Sn — E [Sp] |> t) <2e PG? 


The key to proving Hoeffding's inequality is the following upper bound: if Z is a random variable with 
E|Z| =0Oanda < Z <b, then 
Equation: 


2 (b-a)2 


E [e*7] <e 8 


This upper bound is derived as follows. By the convexity of the exponential function, 
Equation: 


z-a b-—z 
es e? + —— “6 fora < z= b. 
—a 


= 
= 
N 


Convexity of exponential function. 


Thus, 


Equation: 
Z-—a b-Z 
sZ sb sa 
Ele | S 5|5—*|« +B[ |e 
b $a a sb : EZ] 0 
= e€ e€ since = 
b—a b-a ’ 
= (1 —6+ Get-2) | ¢-Sel-a) , where 0= _ 
b-a 

Now let 
Equation: 


u=s(b—a) and define y(u) = —Ou+ log (1 — 6+ Oe"). 


Then we have 
Equation: 


E[e@] < (1 ap a Gert?) ) e-Holb-2) = eP(%), 


To minimize the upper bound let's express y(w) in a Taylor's series with remainder : 
Equation: 


y (u) = (0) + uy’ (0) + 2 gt (v) for some v € {0, ul 


2 
Equation: 
Je" 
; = -6 => y'(u) =0 
gy’ (u) vaespampee = 
be" (de)? 


1—6+ dev (1-6 +e)? 


_ be" 1 be" 
~~ [—6+4+ der 1— 6+ 6er 


= pl =p) 
Now, y” (uw) is maximized by 
Equation: 
a ie 35 WS r 
So, 


Equation: 


Equation: 


Now, we can apply this upper bound to derive Hoeffding's inequality. 
Equation: 


P(S, — E[S,] >t) 
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At 
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by choosing s = 


—2¢ 


Similarly, P(E[S,,] — S, >t) < e*"%" , This completes the proof of the Hoeffding's theorem. 


Example: 
Application 
Let Z; = 1px, zy, -—R (f), as in the classification problem. Then for a fixed f, it follows from Hoeffding's 


inequality (i.e., Chernoff's bound in this special case) that 
Equation: 


P(|R.(f)-R(A|>e) = P(=Is, - BISull > «] 
= P(|\S,—E[Sp]| > ne) 


bet 2(ne)? 


IA 


2e 
= Den 2ne 

Now, we want a bound like this to hold uniformly for all f € ¥. Assume that ¥ is a finite collection of 

models and let |. ¥| denote its cardinality. We would like to bound the probability that 

Max fe F Rn (f)-—R (f)| > e. Note that the event 

Equation: 
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Therefore 


Equation: 


P(max (1) — R(A)| ><) 


°(U IR, (f) - R(A)| >) 


SEF 


Sf Ae, (f) — R(f) |> 6), the **union of events” bound 
fer 
<2 lene. by Hoeffding’s inequality. 


Thus, we have shown that with probability at least 1 — 2|Fle-2"",, Vf € F 
Equation: 


Rn (f) - RUA) <e. 


And accordingly, we can be reasonably confident in selecting f from .F based on the empirical risk function 
R,,. 


Classification Error Bounds 


Recap: Classifier design 


Given a set of training data {X;, Yop and a finite collection of candidate functions F, select fa CF 
that (hopefully) is a good predictor for future cases. That is 
Equation: 


fa =argmin Rr (f) 


where R, (f) is the empirical risk. For any particular f € ¥, the corresponding empirical risk is defined 
as 
Equation: 


as 1 
Rn (f) = — de lyenena- 
i=1 


Hoeffding's inequality 


Hoeffding's inequality (Chernoff's bound in this case) allows us to gauge how close R,, (f) is to the true 
risk of f, R(f), in probability 
Equation: 


P(|Rn (f) — R(f)|> 6) < 267. 


Since our selection process involves deciding among all f € ¥, we would like to gauge how close the 
empirical risks are to their expected values. We can do this by studying the probability that one or more of 
the empirical risks deviates significantly from its expected value. This is captured by the probability 
Equation: 


P(max |, (f) — RUA) > c). 


feF 


Note that the event 
Equation: 


is equivalent to union of the events 
Equation: 


Therefore, we can use Bonferonni's bound (aka the “union of events” or “union” bound) to obtain 
Equation: 


fEF 


P(max | (f)-RWIZ6) = °(U Bn ()- RUA) >+) 
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where |. “| is the number of classifiers in ¥. In the proof of Hoeffding's inequality we also obtained a one- 
sided inequality that implied 
Equation: 


and hence 
Equation: 


P(max R(f) — By (f) > c) < |Fle. 


We can restate the inequality above as follows, For all f € ¥ and for all 6 > 0 with probability at least 
1-6 
Equation: 


Rif) < Bat) +f EA 


2n 


This follows by setting 6 = |.F%|e~2"©’ and solving for e. Thus with a high probability (1 — 6), the true risk 
for all f € F is bounded by the empirical risk of f plus a constant that depends on 6 > 0, the number of 
training samples n, and the size A. Most importantly the bound does not depend on the unknown 
distribution Pyy. Therefore, we can call this a distribution-free bound. 


Error Bounds 
We can use the distribution-free bound above to obtain a bound on the expected performance of the 


minimum empirical risk classifier 
Equation: 


ts =argmin Rr (f). 


We are interested in bounding 


Equation: 


the expected risk of Fr minus the minimum risk for all f € ¥. Note that this difference is always non- 


negative since f,, is at best as good as 
Equation: 


ith =aremin R (f). 


Recall that Vf € FY and V6 > 0, with probability at least 1 — 6 
Equation: 


R(f) < Rn (f) + C(F,n, 8) 


where 
Equation: 


(81,8) = BEF we, 


In particular, since this holds for all f € F including fv 
Equation: 


and for any other f € F 
Equation: 


since R,, (fn) < R, (f)Vf € F. In particular, 


Equation: 


where f* =argminseg R(f). 


Let 2 denote the set of events on which the above inequality holds. Then by definition 
Equation: 


P(2) >1-6. 


We can now bound & [R (7) —R (f°) as follows 
Equation: 
5|R(f)|-R(s") = BIR) - a (F 
= 2|R( 


since E Rn ( f ») =R ( f ps The quantity above is bounded as follows. 


Equation: 


o[R(i.)-R(7)] = 2[R 2 At eB Lele eae) are) 


< B|R 


since P(2) < 1,1— P(2) < dandR (Fn) 2s (f°) <1 


Equation: 
s[e(h)-R(r)|o] < e[a(A) -R (7) 
< C(F,n, 6) 

Thus 

Equation: 

E [R (Fn) Ry (7°) < O(F,n,6) +6. 

So we have 
Equation: 


+ 6, Vd > 0. 


g rE |.F|+ log (1/6) 


2n 


In particular, for 6 = /1/n, we have 


Equation: 
an on log|#|+logn | 1 
= < 
E [R (fn) fer R(f) < Yi In Jn 
log |.F|+ 1 2 = 5 
< (= pre , since Vat+V¥<v2/a+y, ¥ 2,y>0 
n 


Application: Histogram Classifier 


Let F be the collection of all classifiers with M equal volume cells. Then |.F| = 2” 


classification rule 


, and the histogram 


Equation: 
fa =arepin (7901 
n =argmin | — JAY; 
Ore no {f(Xi) AY} 
satisfies 
Equation: 


M log 2+ 2+ log n 
Qa) eee ee 


n 


which suggests the choice M =log, n (balancing M log 2 with log 7), resulting in 


Equation: 
e[R(h)|- pp a =0( 8), 


Error Bounds in Countably Infinite Spaces 


Introduction 


In the last lecture, we studied bounds of the following form: for any 6 > 0, with probability at least 1 — 6, 
Equation: 


i log | F¥|+ log (4 
nin Rain ns , VWhEeF 


which led to upper bounds on the estimation error of the form 
Equation: 


a E (#.)|- min R(f) < = |.F|+ log (n) +2 


n 


The key assumptions made in deriving the error bounds were: 


¢ (i) bounded loss function 
e (ii) finite collection of candidate functions 


The bounds are valid for every Pxy and are called distribution-free . 


Deriving Bounds for Countably Infinite Spaces 


In this lecture we will generalize the previous results in a powerful way by developing bounds applicable to 
possibly infinite collections of candidates. To start let us suppose that ¥ is a countable, possibly infinite, 
collection of candidate functions. Assign a positive number c(f) to each f € F, such that 


Equation: 
S> ef) < 0, 
fEF 


The numbers c(f) can be interpreted as 
e (i) measures of complexity 
e (ii) -log of prior probabilities 
¢ (iii) codelengths 


In particular, if P(f) is the prior probability of f then 
Equation: 


e~ (—logp(f)) = p (f) 


so c(f) = — log p(f) produces 
Equation: 


yevey sei 


SEF SEF 


Now recall Hoeffding's inequality. For each f and every € > 0 
Equation: 


or for every 6 > 0 


Equation: 
BS log (5) 
P — Rn <6 
R(f)- Raf) 27 | < 
Suppose 6 > 0 is specified. Using the values c(f) for f € F, define 
Equation: 
5(f) =e M6, 
Then we have 
Equation: 
1 
los (stp) 
P| R(f) - B,(f) = | — < 4(f). 
n 
Furthermore we can apply the union bound as follows 
Equation: 
log (2 
x log (1/6(f)) zs 8 aA 
P( sup | R(f) —Ra(f)— J = b a0) < PLU R-R(A= eGo) 
SEF n FoF n 
log (ats) 
~ (f) 
< DPR) hae) 
too \ 2n 
< Nan=NeMs=s 
fEF SEF 


So for any 6 > 0 with probability at least 1 — 6, we have that Vf © F 
Equation: 


aH) 
RB, (f) + — 
a log (+ 
= RA+ {score ) (s) 
Special Case 


Suppose F is finite and c(f) =log | ¥| Vf € ¥. Then 


Equation: 
S> as) — ba esl Fl — S> 
fEF fEF SEF ch 
and 
Equation: 
“) 
(N= 
|F | 


which implies that for any 6 > 0 with probability at least 1 — 6, we have 
Equation: 


. log |.F\+ log (+. 
R(f) < Ba(f) + ” (ain)  WPed. 


Note that this is precisely the bound we derived in the last lecture. 
Choosing c(f) 


The generalized bounds allow us to handle countably infinite collections of candidate functions, but we 
require that 
Equation: 


S> eS) < a, 


SEF 
Of course, if c(f) = — log p(f) where p(f) is a proper prior probability distribution then we have 


Equation: 
S> ef) = 1, 
fEeF 


However, it may be difficult to design a probability distribution over an infinite class of candidates. The 
coding perspective provides a very practical means to this end. 


Assume that we have assigned a uniquely decodable binary code to each f € F¥, and let c(f) denote the 
codelength for f. That is, the code for f is c(f) bits long. A very useful class of uniquely decodable codes 
are called prefix codes . 


Prefix Code 
A code is called a prefix code if no codeword is a prefix of any other codeword. 


Example: 
From Cover & Thomas '91 
Consider an alphabet of symbols, say A, B, C’, and D and the codebooks below 


Singular Nonsingular But Not | Uniquely Decodable But] Prefix Code 
Codebook | Uniquely Decodable Not a Prefix Code 


In the singular codebook we assign the same codeword to each symbol - a system that is obviously flawed! 
In the second case, the codes are not singular but the codeword 010 could represent B or CA or AD. Hence 
it is not a uniquely decodable codebook. 

The third and fourth cases are both examples of uniquely decodable codebooks, but the fourth has the 
added feature that no codeword is a prefix of another. Prefix codes can be decoded from left to right since 
each codeword is “self-punctuating" - in this case with a zero to indicate the end of each word. 

To design a uniquely decodable codebook in general is as challenging as the problem of selecting c(f) to 
satisfy 

Equation: 


ef) < a, 


fEF 


However, prefix codes can often be easily designed or specified and they are inherently decodable. 
Moreover, prefix codes satisfy an important inequality called the Kraft Inequality . 


The Kraft Inequality 


For any binary prefix code, the codeword lengths ci, Co, ... satisfy 
Equation: 


Conversely, given any Cj, Ca, ... satisfying the inequality above we can construct a prefix code with these 
codeword lengths. We will prove this result a bit later, but now let's see how this is useful in our learning 
problem. 


Assume that we have assigned a binary prefix codeword to each f € F¥, and let c(f) denote the bit-length 
of the codeword for f. Set 6(f) = 2~°)6. Then 
Equation: 


LU R(f) — Rn (f) = vee (xin) 


fEF 2n = 2° R(f) — Rn(f) = 5 
< Safar = 
fEF fEF 


This implies that for any 6 > 0 with probability at least 1 — 6 we have Vf € F 
Equation: 


Application 

Let F\, Fi, ... be a sequence of finite sets of candidate functions with |. F; |<|.4A1|<... We can design 
prefix codes as follows. Use the codes 0, 10, 110, 1110, ... to encode the subscript 7 in | F;|. For each class 
[log, |.F|| to uniquely encode each function in F;. 
Then, encode any given function f by first using the code for 2 corresponding to the smallest F; that f 
belongs to, followed by the length [log, |.F|| codeword for f € F;. This is a prefix code. 


Example: 
Histogram Classifiers 
X=[0,1]¢, Y={0,1}. Let F,, k=1, 2, ... denote the collection of histogram classification rules with k equal 


volume bins. We can use the following codebook for the index k. 
[__k | Prafix Code _| 
a a a 


And follow this codeword with k =log,|.F,| bits to indicate which of the 2* possible histogram rules is 


under consideration. Thus for any f € Y, for some k > 1 there is a prefix code of length 
Equation: 


ce(f) =k+k=2k bits. 
It follows that for any 6 > 0 with probability at least 1 — d we have Vf € Uys1 Fk 


Equation: 


z 2k log 2+ | 4 
soo Gn Ce 


where ky is the number of bins in histogram corresponding to f. Contrast with the bound we had for the 
class of m bin histograms alone: with probability > 1 — 6, Vf € Fn 


Equation: 


m log 2+ log (ar) 
2n 


Notice the bound for all histograms rules is almost as good as the bound for only the m-bin rules. That is, 
when k f=m the bounds are within a factor of a7 Ds On the other hand, the new bound is a big 
improvement, since it also gives us a guide for selecting the number of bins. 

Proof 

Proof of the Kraft Inequality 


We will prove that for any binary prefix code, the codeword lengths c, €2, ... satisfy }),., 2- < 1. The 


converse is easy to prove also, but it not central to our purposes here (for a proof, see Cover & Thomas 
'91). Consider a binary tree like the one shown below. 


ooo 001 


The sequence of bit values leading from the root to a leaf of the tree represents a codeword. The prefix 
condition implies that no codeword is a descendant of any other codeword in the tree. Let Cjyaz be the 
length of the longest codeword (also the number of branches to the deepest leaf) in the tree. 


Consider a leaf 2 in the tree at level c;. This leaf would have 27% descendants at level Cmax. 
Furthermore, for each leaf the set of possible descendants at level Cmaz is disjoint (since no codeword can 
be a prefix of another). Therefore, since the total number of possible leafs at level Cngz is 2°", we have 
Equation: 


yy 2Cmax—Ci eS 2emaz = SS 2% <1 


i€leafs i€leafs 


which proves the case when the number of codewords is finite. 
Suppose now that we have a countably infinite number of codewords. Let b; bg ... be, be the ith codeword 


and let 
Equation: 


Ty = > b;279 
jat 


be the real number corresponding to the binary expansion of the codeword. We can associate the interval 
[r;,7; + 2-%) with the ith codeword. This is the set of all real numbers whose binary expansion begins 
with b; bg ... be,. Since this is a subinterval of {0, 1], and all such subintervals corresponding to prefix 
codewords are disjoint, the sum of their lengths must be less than or equal to 1. This proves the case where 
the number of codewords is infinite. 


Complexity Regularization 


Review: PAC Bounds 


Consider a finite collection of models ¥, and recall the basic PAC bound: for any 6 > 0, with probability 
at least 1 — 6 


Equation: 
RUN SR (p+y EHO) | ves 
where 
Equation: 
Ba) = ete)" 


Rf) = Elef(X),¥)I 


and the loss £ is assumed to be bounded between 0 and 1. Note that we can write the inequality above as: 


Equation: 
: log (471) 
R(f) < Rn (f) + aa 
Letting 65 = TA we have: 
Equation: 
‘ log (1/5) 
R(f) s Ralf) + 4{— Jo 


This is precisely the form of Hoeffding's inequality, with 6 in place of the usual 4. In effect, in order to 
have Hoeffding's inequality hold with probability 1 — 6 for all f € Y, we must distribute the “d-budget” 
or “confidence-budget” over all f € F (in this case, evenly distributed): 

Equation: 


6 

re) —— 

248 = is 
= ¢ 


However, to apply the union bound, we do not need to distribute 6 evenly among the candidate models. 
We only require: 
Equation: 


yop = 5 


fEF 


So, if p(f) are positive numbers satisfying }/ ¢<.gp(f) = 1, then we can take 6 = p(f)6. This 
provides two advantages: 


1. By choosing p(f) larger for certain f, we can preferentially treat those candidates 
2. We do not need F to be finite and we only require 7 ;.gp(f) = 1 


Prefix codes are one way to achieve this. If we assign a binary prefix code of length c(f) to each f € F, 
then the values p(f) = 2~°) satisfy > tex P(f) < 1 according to the Kraft inequality. 


The main point of this lecture is to examine how PAC bounds of the form w.p. > 1 — 6 
Equation: 


RUA) < R(t) + of Eee 2 We » eee 


can be used to select a model that comes close to achieving the best possible performace 
Equation: 


int RUD 


Let f;, be the model selected from F using the training data {X;, Y;};_,. We will specify this model in a 
moment, but keep in mind that it is not necessarily the model with minimum empirical risk as before. We 
would like to have 

Equation: 


a[r(7))- js aun 


as small as possible. First, for any 6 > 0, define 


Equation: 
ry : D 
fi = aremin {Rn (f)+C (F.n,9)} 
fEF 
where 
Equation: 


C(f,n,8) = (2 log “tts (1/6) 


Then w.p. > 1-6 
Equation: 


R(f) <BR, (f)+C(F,n,6) , VESEF 


and in particular, 
Equation: 


so, by the definition of Ff, Vf EF 
Equation: 


R (fi) < R,(f)+C(f,n,68) . 


We will make use of the inequality above in a moment. First note that Vf € F 
Equation: 


B(R(f)|-RU) = B[R(f)- BN] +2 [R()- RB) 


The second term is exactly 0, since E Re, ()| = Af). 


Now consider the first term [R ig} Ry ( f)| . Let (2 be the set of events on which 


Equation: 


R( fr) < Ba(f) —C(fimsd), V FEF 


From the bound above, we know that P(2) > 1 — 6. Thus, 
Equation: 


B|R (fa) -Ra(f| = #[R(ft) - Ba (f)|0]P (2) +2[R (fa) — Ba (4) 2"] (1 - P(O)) 
C(f,n,6) +6 (since 0 < R, R<1, P(2) < Land 1~ P(2) <8) 
7 ye) 


2n 


IA 


+6 


lop: 244 1 
_ ,[clloe2+zlogn 1 (vy setting = 1) 


We can summarize our analysis with the following theorem. 
Theorem 
Complexity Regularized Model Selection 


Let F be a countable collection of models, and assign a positive number c(f) to each f € F such that 
y feF 2-f) < 1. Define the minimum complexity regularized risk model 


Equation: 
zs ee o(f) og 2+ $ los | 
f argmin J (f) + = 

Then, 

Equation: 


s[r(h)] < py ain+ (ert been 1 


This shows that 
Equation: 


1 
A+ y aed ln 


is a reasonable surrogate for 
Equation: 


0 
nine log cis x logn 


Example: 

Histogram Classifiers 

Let 2% = (0, 1]% be the input space and Y = {0,1} be the output space. Let Fy, k = 1, 2, ... denotes 
the collection of histogram classification rules with k equal volume bins. One choice of prefix code for 
this example is: k = 1 => code = 0,k = 3 > code = 10, k = 3 => code = 110 and so on.... Then, if 
first code is corresponding to k > f € Fx, followed by k =log, |-F,| bits to indicate which of the 2" 
histogram rules in Yj, is under consideration, we have 

Equation: 


f € Fx => c(f) = 2k bits 


Let f, be the model that solves the minimization i.e., 
Equation: 


a 2k log 2+ 4 log n 
: in B 2 
Bt en ON oe 


That is, for each k, let 
Equation: 


atk) in B 
th ope n(f) 


Then select the best & according to 


Equation: 
Zs a 2k log2++logn 
k = argmin < R, () + 2 zoe 
k>1 2n 
and set 
Equation: 
A k 
in = fA ) 
Then, 
Equation: 


E|R(fa)| < inf ( min R(f)+ 


2k log 2+ + logn i 1 


2n Jn 


It is a simple exercise to show that if d = 2 and the Bayes decision boundary is a 1-d curve, then by 
setting k = \/n and selecting the best f from F vn We have 


Equation: 


Note:The complexity regularized classifier f,, adaptively achieves this rate, without user intervention. 


Decision Trees 


Minimum Complexity Penalized Function 


Recall the basic results of the last lectures: let 2 and Y denote the input and output 
spaces respectively. Let X € 2 and Y € & be random variables with unknown 
joint probability distribution Pxyy. We would like to use X to “predict” Y. Consider a 
loss function 0 < (yi, y2) <1, Vy1, y2 € &. This function is used to measure the 
accuracy of our prediction. Let FY be a collection of candidate functions (models), 
f: & — Y. The expected risk we incur is given by R(f) = Exy |€(f (X), Y))]. 
We have access only to a number of i.i.d. samples, {X;, Yi} os These allow us to 
compute the empirical risk 2, (f) = 4 EF (Xi), Yi). 

Assume in the following that F is countable. Assign a positive number c(f) to each 
f € F such that peg 2-<(f) < 1. If we use a prefix code to describe each element 
of F and define c(f) to be the codeword length (in bits) for each f € F, the last 


inequality is automatically satisfied. 


We define the minimum complexity penalized estimator as 
Equation: 


= Ee c(f) log 2+ + logn 
n= 1 ie en 
f argmin (f) + a 


As we showed previously we have the bound 
Equation: 


| Onesie 1 


B|R (Fn) <min ¢ R(f) 4 os a 


fEF 


The performance (risk) of a is on average better than 
Equation: 


nS c(f,) log 2+ 4 logn 1 
R(t) | / 2n Jn’ 


where 
Equation: 


* o(f) og 2+ + log | 
f, =argmin [n+ oes Ee) 


If it happens that the optimal function, that is 
Equation: 


f =arg min R(f), 


f measurable 


is close to an f € ¥ witha small c(f), then fo will perform almost as well as the 
optimal function. 


Example: 
Suppose f. € F, then 
Equation: 


E|R(f,)| 2h) 


2n Jn 


Furthermore if c (f~) = O (log n) then 


e[R(f)] <2(s*) +0( =") 


that is, only within a small O (y/ ee | offset of the optimal risk. 


In general, we can also bound the excess risk [R Ge) — R*, where R’ is the 


Bayes risk, 
Equation: 


= R(f). 

if Pear te (f) 
By subtracting R* (a constant) from both sides of the inequality 
Equation: 


E [R (Fn) <min ¢ R(f) + jedi aealie =a = AUS at - 


we obtain 
Equation: 


E|R(fa)| — R* <min ery 


Note that two terms in this upper bound: R(f) — R’ is a bound on the 
approximation error of a model f, and remainder is a bound on the estimation error 
associated with f. Thus, we see that complexity regularization automatically 
optimizes a balance between approximation and estimation errors. In other words, 
complexity regularization is adaptive to the unknown tradeoff between 
approximation and estimation. 


Classification 


Consider the particularization of the above to a classification scenario. Let 
# =(0,1]%, Y = {0,1} and £ (j,y) = 1 pp zy- Then 

R(f) = Exy [1lxzvy] = P (f (X) 4 Y). The Bayes risk is given by 
Equation: 


R°= inf R(f). 


f measurable 


As it was observed before, the Bayes classifier (i.e., a classifier that achieves the 
Bayes risk) is given by 
Equation: 


wl plR 


This classifier can be expressed in a different way. Consider the set 
G’ ={x: P(Y =1|X =z) > 1/2}. The Bayes classifier can written as 
f : (@)=1 {ecG"}- Therefore the classifier is characterized entirely by the set G “wif 


X € G then the “best” guess is that Y is one, and vice-versa. The boundary of this 
set corresponds to the points where the decision is harder. The boundary of G” is 
called the Bayes Decision Boundary. In [link](a) this concept is illustrated. If 

n(x) = P(Y = 1|X = 2) is a continuous function then the Bayes decision boundary 
is simply given by {a : P(Y =1|X = x) = 1/2}. Clearly the structure of the 
decision boundary provides important information on the difficulty of the problem. 


(a) The Bayes classifier and the Bayes decision boundary ; (b) Example of the 
Li.d. training pairs. 


¥ = [0,17 


Bayes Decision 
Boundary 


fin=o 


Bayes Decision 
Boundary 
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Empirical Classifier Design 


Given n iid. training pairs, {X;, Vi} a: we want to construct a classifier f,, that 
performs well on average, i.e., we want [R ( in) | as close to R’ as possible. In 


[link](b) an example of the i.i.d. training pairs is depicted. 


The construction of a classifier boils down to the estimation of the Bayes decision 
boundary. The histogram rule, discussed in a previous lecture, approaches the problem 
by subdividing the feature space into small boxes and taking a majority vote of the 
training data in each box. A typical result is depicted in [link](a). 


The main problem with the histogram rule is that it is solving a more complicated 
problem than it is actually necessary. We do not need to determine the correct label for 
each individual box directly (the histogram rule is essentially estimating n()). In 
principle we only need to locate the decision boundary and assign the correct label on 
either side (notice that the accuracy of a majority vote over a region increases with the 
size of the region). The next example illustrates this. 


Example: 
Three Different Classifiers 


The pictures below correspond to the approximation of the Bayes classifier by three 
different classifiers: 


(a) Histogram classifier ; (b) Linear classifier; (c)Tree classifier. 


Histogram Classificr 


Tree Classilier 


The linear classifier and the tree classifier (to be defined formally later) both attack 
the problem of finding the boundary more directly than the histogram classifier, and 
therefore they tend to produce much better results in theory and practice. In the 
following we will demonstrate this for classification trees. 


Binary Classification Trees 
Binary classification trees are constructed by a two-step process: 


1. Tree growing 
2. Tree pruning 


The basic idea is to first grow a very large, complicated tree classifier, that explains 
the the training data very accurately, but has poor generalization characteristics, and 
then prune this tree, to avoid overfitting. 


Growing Trees 


The growing process is based on recursively subudividing the feature space. Usually 
the subdivisions are splits of existing regions into two smaller regions (i.e., binary 
splits) and usually the splits are perpendicular to one of the feature axes. An example 
of such construction is depicted in [link]. 


| . ia . | | . ie a 


Growing a recursive binary tree (2 = [0, 1] a 


Often the splitting process is based on the training data, and is designed to separate 
data with different labels as much as possible. It such constructions, the “splits,” and 
hence the tree-structure itself, are data dependent. Alternatively, the splitting and 
subdivision could be independent from the training data. The latter approach is the 
one we are going to investigate in detail, and we will consider Dyadic Decision Trees 
and Recursive Dyadic Partitions (depicted in [link]) in particular. 


Until now we have been referring to trees, but did not make clear how do trees relate 
to partitions. It turns out that any decision tree can be associated with a partition of the 
input space 2 and vice-versa. In particular, a Recursive Dyadic Partition (RDP) can 
be associated with a (binary) tree. In fact, this is the most efficient way of describing a 


RDP. In [link] we illustrate the procedure. Each leaf of the tree corresponds to a cell 
of the partition. The nodes in the tree correspond to the various partition cells that are 
generated through in the construction of the tree. The orientation of the dyadic split 
alternates between the levels of the tree (for the example of [link], at the root level the 
split is done in the horizontal axis, at the level below that (the level of nodes 2 and 3) 
the split is done in the vertical axis, and so on...). The tree is called dyadic because the 
splits of cells are always at the midpoint along one coordinate axis, and consequently 
the sidelengths of all cells are dyadic (i.e., powers of 2). 
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Example of Recursive Dyadic Partition (RDP) growing (2% = [0, 1] *, 


In the following we are going to consider the 2-dimensional case, but all the results 
can be easily generalized for the d-dimensional case (d > 2), provided the dyadic tree 
construction is defined properly. Consider a recursive dyadic partition of the feature 
space into k boxes of equal size. Associated with this partition is a tree T. Minimizing 
the empirical risk with respect to this partition produces the histogram classifier with 
k, equal-sized bins. Consider also all the possible partitions corresponding to pruned 
versions of the tree 7’. Minimizing the empirical risk with respect to those other 
partitions results in other classifiers (dyadic decision trees) that are fundamentally 
different than the histogram rule we analyzed earlier. 


Pruning 


Let F be the collection of all possible dyadic decision trees corresponding to 
recursive dyadic partitions of the feature space. Each such tree can be prefix encoded 


with a bit-string proportional to the number of leafs in the tree as follows; encode the 
structure of the tree in a top-down fashion: (i) assign a zero at each branch node and a 
one at each leaf node (terminal node) (ii) read the code in a breadth-first fashion, top- 
down, left-right. [link] exemplifies this coding strategy. Notice that, since we are 
considering binary trees, the total number of nodes is twice the number of leafs minus 
one, that is, if the number of leafs in the tree is k then the number of nodes is 2k — 1. 
Therefore to encode a tree with k leafs we need 2k — 1 bits. 


Since we want to use the partition associated with this tree for classification we need 
to assign a decision label (either zero or one) to each leaf. Hence, to encode a decision 
tree in this fashion we need 3k — 1 bits, where k is the number of leafs. For a tree 
with k leafs the first 2k — 1 bits of the codeword encode the tree structure, and the 
remaining k bits encode the classification labels. This is easily shown to be a prefix 
code, therefore we can use this under our classification scenario. 


Illustration of the tree 


coding technique: example 
of a tree and corresponding 
prefix code. 


Let 
Equation: 


(3k — 1) log 2+ 5 logn 
2n 


This optimization can be solved through a bottom-up pruning process (starting from a 
very large initial tree Ty) in O(|T |”) operations, where |T7p| is the number of leafs in 
the initial tree. The complexity regularization theorem tells us that 


Equation: 


; rn (3k — 1) log 2+ > logn Fi 1 
~ feF 2n Jn 


Comparison between Histogram Classifiers and Classification Trees 


In the following we will illustrate the idea behind complexity regularization by 
applying the basic theorem to histogram classifiers and classification trees (using our 
setup above). 


Consider the classification setup described in "Classification", with 2 = [0, i. 


Histogram Risk Bound 
Recall the setup and results of a previous lecture[footnote]. Let 


The description here is slightly different than the one in the previous lecture. 
Equation: 


FF = {histogram rules with k? bins}. 


Then FF |= 2” Let FY = Uyo1 Fi". We can encode each element f of A” 


with cy (f) = k + k? bits, where the first k bits indicate the smallest k such that 
fEF oa and the following k? bits encode the labels of each bin. This is a prefix 
encoding of all the elements in F 7. 


We define our estimator as 


Equation: 
jt Fi 
where 
Equation: 
fr? =arg min, R,, (f), 


and 
Equation: 


= = k+k?) log 2+ + logn 
k =argmin Ry Ff? + (ee 
k>1 2n 


Therefore f// minimizes 
Equation: 


cy (f) log 2+ +logn 


2n j 


over all f € ##. We showed before that 
Equation: 


cy (f) log 2+ 4 logn 1 
a ne 
fEeF# 2n Jn 


To proceed with our analysis we need to make some assumptions on the intrinsic 
difficulty of the problem. We will assume that the Bayes decision boundary is a “well- 
behaved” 1-dimensional set, in the sense that it has box-counting dimension one (see 
Appendix "Box Counting Dimension"). This implies that, for an histogram with k? 
bins, the Bayes decision boundary intersects less than C’k bins, where C’ is a constant 
that does not depend on k. Furthermore we assume that the marginal distribution of X 
satisfies Px (A) < K|A\, for any measurable subset A C (0, 1]”. This means that the 
samples collected do not accumulate anywhere in the unit square. 


Under the above assumptions we can conclude that 
Equation: 
CK 


min R(f)—R  < Oe —. 
fare k? k 


Therefore 


Equation: 


E|R(f#)|-R < oxja+ ete) oea bogn | 1 


We can balance the terms in the right side of the above expression using k = n!/4 (for 
n large) therefore 
Equation: 


Dyadic Decision Trees 


Now let's consider the dyadic decision trees, under the assumptions above, and 
contrast these with the histogram classifier. Let 
Equation: 


Fj. = {tree classifiers with k leafs}. 


Let FZ? =U gi 7) 7. We can prefix encode each element f of A* with 
cr (f) = 3k — 1 bits, as described before. 


Let 
Equation: 
oT “os k 
tn = AA ) 
where 
Equation: 
Ak) _—-— 
n  =argmin R, (f), 
f gmin (f) 
and 


Equation: 


a 


a 3k —1)log2++logn 
k =argmin ¢ R, F ( ) log 2 8 
k>1 2n 


Hence fr minimizes 
Equation: 


1 
R,, (f) + \2 W) “et 2 = = ’ 


over all f © #*. Moreover 
Equation: 


u|r(f)|-R < min R(f) es oes a 


If the Bayes decision boundary is a 1-dimensional set, as in "Histogram Risk Bound", 
there exists a tree with at most 8C’k leafs such that the boundary is contained in at 
most Ck squares, each of volume 1/k?. To see this, start with a tree yielding the 
histogram partition with k? boxes (i.e., the tree partitioning the unit square into k? 
equal sized squares). Now prune all the nodes that do not intersect the boundary. In 
[link] we illustrate the procedure. If you carefully bound the number of leafs you need 
at each level you can show that you will have in total less than 8C’k leafs. We 
conclude then that there exists a tree with at most 8C’k leafs that has the same risk as a 
histogram with O(k?) bins. Therefore, using [link] we have 


Equation: 


(3 (8Ck) — 1) log 2+ 4 logn 1 
ae ee a eee ee ee 
2n Jn 


n 


E|R(f*)| i OK ee 


We can balance the terms in the right side of the above expression using k = n‘/3 (for 
n large) therefore 
Equation: 


Illustration of the tree pruning procedure: (a) Histogram classification rule, for a 
partition with 16 bins, and corresponding binary tree representation (with 16 
leafs). (b) Pruned version of the histogram tree, yielding exactly the same 
classification rule, but now requiring only 6 leafs. (Note: The trees where 
constructed using the procedure of Figure ) 
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Final Comments 


Trees generally work much better than histogram classifiers. This is essentially 
because they provide much more efficient ways of approximating the Bayes decision 
boundary (as we saw in our example, under reasonable assumptions on the Bayes 


boundary, a tree encoded with O(k) bits can describe the same classifier as an 
histogram that requires O(k?) bits). 


The dyadic decision trees studied here are different than classical tree rules, such as 


CART or C4.5. Those techniques select a tree according to 
Equation: 


es Ce cas 


for some a > 0 whereas ours was roughly 
Equation: 


k =een {Rr (a) =e avi, 


fora = / SoHE The square root penalty is essential for the risk bound. No such 


bound exists for CART or C4.5 . Moreover, recent experimental work has shown that 
the square root penalty often performs better in practice. Finally, recent results show 
that a slightly tighter bounding procedure for the estimation error can be used to show 
that dyadic decision trees (with a slightly different pruning procedure) achieve a rate 
of 

Equation: 


which turns out to be the minimax optimal rate (i.e., under the boundary assumptions 
above, no method can achieve a faster rate of convergence to the Bayes error). 


Box Counting Dimension 


The notion of dimension of a sets arises in many aspects of mathematics, and it is 
particularly relevant to the study of fractals (that besides some important applications 
make really cool t-shirts). The dimension somehow indicates how we should measure 
the contents of a set (length, area, volume, etc...). The box-counting dimension is a 
simple definition of the dimension of a set. The main idea is to cover the set with 
boxes with sidelength r. Let N(r) denote the smallest number of such boxes, then the 
box counting dimension is defined as 


Equation: 


_ log N(r) 
lim —-———. 
r>0 — logr 


Although the boxes considered above do not need to be aligned on a rectangular grid 
(and can in fact overlap) we can usually consider them over a grid and obtain an upper 
bound on the box-counting dimension. To illustrate the main ideas let's consider a 
simple example, and connect it to the classification scenario considered before. 


Let f : [0,1] — [0, 1] be a Lipschitz function, with Lipschitz constant L (i.e., 
|f(a) — f(b)| < Lja— 6], Va,b € (0, 1]}). Define the set 
Equation: 


A= {x = (21,22): %2 = f (z1)}, 


that is, the set A is the graphic of function f. 


Consider a partition with k? squared boxes (just like the ones we used in the 
histograms), the points in set A intersect at most C’k boxes, with C’ = (1+ [L]) 
(and also the number of intersected boxes is greater than k). The sidelength of the 
boxes is 1/k therefore the box-counting dimension of A satisfies 

Equation: 


ae log C'k 
1 —————————— 
1/k>0 — log (1/k) 

/ 
ee log C"+ log (k) 
k—00 log (k) 
1. 


| 


I 


| 


The result above will hold for any “normal” set A C [0, 1] * that does not occupy any 
area. For most sets the box-counting dimension is always going to be an integer, but 
for some “weird” sets (called fractal sets) it is not an integer. For example, the Koch 
curve has box-counting dimension log (4) / log (3) = 1. 26186.... This means that it 
is not quite as small as a 1-dimensional curve, but not as big as a 2-dimensional set 
(hence occupies no area). 


To connect these concepts to our classification scenario consider a simple example. 
Let n(x) = P(Y = 1|X = 2) and assume 7(z) has the form 


Equation: 


n(x) = = +22 - f(a), Va = (@1,22) € &, 


where f : [0,1] — [0, 1] is Lipschitz with Lipschitz constant DL. The Bayes classifier 
is then given by 
Equation: 


* 


Ff (#) = Lgyeyzt/2} = Lfer>s(e1)}- 


This is depicted in [link]. Note that this is a special, restricted class of problems. That 
is, we are considering the subset of all classification problems such that the joint 
distribution Pyy satisfies P(Y = 1|X = x) = 1/2+ 2,4 — f (x1) for some function 
f that is Lipschitz. The Bayes decision boundary is therefore given by 

Equation: 


A=Sde= (24, to) 23 = fF (21)}. 


Has we observed before this set has box-counting dimension 1. 


fix.) - Bayes Decision Boundary 


Bayes decision boundary for the 
setup described in Appendix . 


Complexity Regularization for Squared Error Loss 


Complexity Regularization in Regression 


Recall the classification problem. In Lecture 6, where we assumed that min seg R(f) = 0, we obtained the 
PAC bound Vf € F 
Equation: 


PR (fn) > e} < |Fle-™. 


From Corrolary_1 in Lecture 6, 
Equation: 


be(j)] < me 


n 


In Lectures 7 and 8, we dropped the assumption that min s-g R(f) = 0 and obtained, Vf € F 
Equation: 


PR (Fn) > e} <|Fle". 


This led to 
Equation: 


B|R (fn) — min RA) < eee 


fEF n 


Hoeffding's inequality was central to our analysis of learning under bounded loss functions. In many 
regression and signal estimation problems it is natural to consider squared error loss functions (rather than 
0/1 or absolute error). In such cases, we will need to derive bounds using different techniques. 


Example: 
To illustrate the distinction between classification and regression, consider a simple, scalar signal plus noise 
problem. Consider Y; = 6 + W;,7 = 1,---,n, where @ is a fixed unknown scalar parameter and the W; are 


independent, zero-mean, unit variance random variables. Let Y=1 /n Se Y;. Then, according to the 


Central Limit Theorem, Y is distributed approximately N(6, 1/n). A simple tail-bound on the Gaussian 
distribution gives us 
Equation: 


= il ae 
P(¥-0>c)=P(W>e) ge 


which implies that 
Equation: 


PUG en en 


This is a bound on the deviations of the squared error err? = | Y— 6| = Notice that the exponential decay 
rate is a function of € rather than €?, as in Hoeffding's inequality. The squared error concentration inequality 
implies that E[|Y — 6|?] = (0) (+) (just write & [err?] = lee IP (err? > t) dt). Therefore, in regression 
with a squared error loss, we can hope to get a rate of convergence as fast as n instead of n~'/?, The 
reason is simply because we are using an squared error loss instead of the 0/1 or absolute error loss. 

To begin our investigation into regression and function estimation, let us consider the following. Let 


XK =R4 and Y = R. Take ¥Fsuch that f € F isamap f : R? +> R. We have training data 
iid. 


{X;, Yi}, ~  Pxy. As our loss function, we take the squared error, i.e., 
Equation: 
2 
l(f (Xi), Yi) aa (f (Xi) . 1s) 
The risk is then the MSE: 
Equation: 


We know that the function f * that minimizes the MSE is just the conditional expectation of Y given X: 
Equation: 


f =E\Y|X =a. 
Now let R' =R ( i oe We would like to select an fa € F using the training data {X;, Y;};"_, such that the 


excess risk 
Equation: 


E|R(fa)| 8 Si 


is small. Let's consider the difference between the empirical risks: 
Equation: 


R(n-R(f') == or) —¥)? - 


Note that £7 [z (f) - R (f*)| =R(f)-R Ge Hence, by the Strong Law of Large Numbers (SLLN), 


we know that 
Equation: 


R(f)-R(¢") > R()-R(F) 


as n — oo. But how fast is this convergence? 
We will derive a PAC style bound for the difference R(f) — R ( i | - (R (f) —R ( f ae The following 


derivation is from Barron 1991. The excess risk and it empirical counterpart will be denoted by 
Equation: 


Note that 7 ( Hy df ) is the sum of independent random variables: 
Equation: 


‘(nf)--Z ou 


where U; = —(Y; — f (X;))? + (Yi — ip Coin Therefore, 


ele] Die (Oe EE) 
We are looking for a PAC bound of the form 


Equation: 
P(r (ae) 38 (Cae) = c) 26; 


If the variables U; are bounded, then we can apply Hoeffding's inequality. However, a more useful bound 
for our regression problem can be derived if the the variables U; satisfy the following moment condition: 
Equation: 


E||U; — E [Ui] ape 


k var(U;) 


for some h > 0. 

The moment condition can be difficult to verify in general, but it does hold, for example, for bounded 
random variables. If [link] holds, then the Craig-Bernstein (CB) inequality states: 

Equation: 


iy ne var(+ S°U; 
9(2 3-210) > ~ 4: ae ') Se", 


=I 


for 0 < €h <c < landt > 0. This shows that the tail decays exponentially in t, rather than exponentially 
in t?. Recall Hoeffding's inequality: 
Equation: 


42 


If + < 1, then = < t, which implies e >> e '. This indicates that the CB inequality may be much 
tighter than Hoeffding's. To use the CB inequality, we need to bound the variance of = oe, Ui. Note that 
Equation: 


var (Ui) = war ( —0% - £26)" + (¥- #° @%))’). 


Assumption 1 
The support of Y and the range f(X) is in a known interval of length b. 


Proposition 1 
2 
With the above assumption, [link] holds with h = a 


Proposition 2 
Again, with the above assumption, it may be shown that 
Equation: 


var (U;) < 5b?r (/, a 


You can write U; as 
Equation: 


* 


U; = 2; f (Xi) — 2¥if" (Xi) + FG)? - FG)’ 
= 2Y;,f (X;) — 2Y,f* (X;) + 2f"(Xi)? — f (Xi)? — f(%)? + 2F (Xi) fF" (Xi) — 2F (Xi) fF" (X,), 


= 2(¥i- F(x) (F(R) — FD) - (FR) — FH)’ 


( 
a 


Note that the variance of U; is upper-bounded by its second moment. Also note that the covariance of the 
two terms above is zero: 
Equation: 


B[2( - #° (x) (10) - F(X) (F00)— FD)" | = BITTY 
= Ex [Ey\x [MT] 
= Ex [T,Ey\x (T]| 
Ex |T,*0| 
= 0 


This is evident when you recall that f° (X;) = E[Y|X = X;]. Now we can bound the second moments of 
T; and T» 4 
Equation: 


ET] = 4E 


E[T)] = E 


IA 
& 


So var (U;) < 5b’ E I(F (w=ouf (X;)) " . The final step is to see that 


Equation: 


6 (2°) = BW = Bx [Ex] = 8 [(s00) 4 xa)’ ]. 


Thus, n var (= eee, U;) < 5b?r (f, a) . And therefore, we can say that, with probability at least 1 — e~’, 
Equation: 


OH) (Ah )se ones 


In other words, with probability at least 1 — 6 (where 6 = e~*), 
Equation: 


r(ne)-F(t4) < a he ene 


Now, suppose we have assigned positive numbers c(f) to each f € F satisfying the Kraft inequality: 


Equation: 
>» 9-cf) = 1, 
SEF 


Note that [link] holds V6 > 0. In particular, we let 6 be a function of f: 
Equation: 
5(f) = 2-6, 
So we can use this 6 along with the procedure introduced in Lecture 9 (i.e., Union of events bound followed 


by the Kraft inequality) to obtain the following. For all f € ¥,V6 > 0, 
Equation: 


f 7 : c(f)log2+log = Be B? PUT ) 
r(tF)-#(BF') s n € : 2(1 —c) 


with probability at least 1 — 6. Nowsetc =€ h= — £ and assume € < Te Then define 
Equation: 

Be B? E 

~ 2(1—c) 


Now, after using @ and rearranging terms, we have: 
Equation: 


c(f) log 2+ log + 


en 


(1-a)r (F,F°) <#(4,F°) + 


We want to choose f to minmize this upper bound. So take 
Equation: 


jy aremin { R, (7) + Wes? | 


ne 


So, with probability at least 1 — 6, 
Equation: 


c (Fn) log 2+ log $ 


(1-a)r (fa, f’) Z *(fuf’) + . 
< *(t.f") n c(f,.) = log 4 


where f. =argmin fez {R (f) + a \. 
Now we use the Craig-Bermstein inequality to bound the difference between 7 ( ee ‘) and r( Teak *) : With 


probability at least 1 — 6, 
Equation: 


(ir) <e(tr)tar(sr)+ ee. 


ne 


Now we can again use the union bound to combine [link] and [link]: With probability at least 1 — 26,Vd6 > 0, 
Equation: 


1+ 
1— 


) " c(f,) log 2+ 2 log 1/6 


tof) saat Fol > 


—net 
Now set 6 = e 2 _, then we have 
Equation: 


9 (>(Iuf’) - te, (, g) 4 Aha) oe >) <2, 


Integrating, we get 
Equation: 


0 


, 2e os 
0 
4 


ne 


Bl (fut) - 2, (f°) 1 hades? < [a i> t) dt 


IA 


To sum up, we have shown that for e < — © 


‘196? ? 
Equation: 
ss l+a « x e(f,) log 2+4 
B[r(AF)] < (4 }r (sas) + ’ 
—a ne 
Or, 
Equation: 


elo) = (728) pp ror) Seep 


since a < 1. Or, in expanded form: 
Equation: 


e[r(h)]-R(F) < (GEE) mp {Rn—a(r) +S he 


Notice that if f° € F and if oF) is not too large (e.g., e(f’) log n), then we have 


E [R (7) —R ( f 2 =O (n“! log n), within a logarithmic factor of the parametric rate of convergence! 


Maximum Likelihood Estimation 


In the last lecture we derived a risk (MSE) bound for regression problems; i.e., select an f € F 
so that E l(f (X) - y)?| —E Gs (X) —Y) " is small, where f° (x) = E [Y|X = a]. The 
result is summarized below. 


Theorem 
Complexity Regularization with Squared Error Loss 


Let & = R4, Y = [—b/2, b/2], {Xi, Yi };_, iid, Pxy unknown, ¥ = {collection of candidate 
functions}, 
Equation: 


frR' oy, R(f)=E|(f(X)-Y)’. 


Let c(f), f € F, be positive numbers satisfying 5> FoF Q-(f) < 1, and select a function from 


F according to 
Equation: 


fn =argmin 4 R, (f) + — 
€ n 


re fr 1 i ios? 


with e < 3; and Rn (f) = + 0”, (f (X;) — Y;)”. Then, 


Fquaone : 
#[R(A)]-(F) = (G25) mp fe-a(F) + Sf oe 
where a = Sa 


Maximum Likelihood Estimation 


The focus of this lecture is to consider another approach to learning based on maximum 
likelihood estimation. Consider the classical signal plus noise model: 
Equation: 


v= (=) + Waist, 
n 


where W; are iid zero-mean noises. Furthermore, assume that W; ~ P (w) for some known 
density P(w). Then 
Equation: 


1-(v-s(5)) =P 


since Y; — f(+) = W;. 
A very common and useful loss function to consider is 
Equation: 
=~ 1 wy 
Bn (f) = — D> (~ log Py, (¥i). 
1 


Minimizing R,, with respect to f is equivalent to maximizing 


Equation: 
LS tog Py (¥;) 
no Sif 

or 

Equation: 


Thus, using the negative log-likelihood as a loss function leads to maximum likelihood 
estimation. If the W; are iid zero-mean Gaussian r.v.s then this is just the squared error loss we 
considered last time. If the W; are Laplacian distributed e.g. P (w) « e |’, then we obtain the 
absolute error, or £1, loss function. We can also handle non-additive models such as the Poisson 
model 

Equation: 


Yi ~ Pylfli/n)) =e Me) LET 


y! 


In this case 
Equation: 


— log P(Y;|f(é/n)) = fli/n) — Y; log (f(é/n)) + constant 


which is a very different loss function, but quite appropriate for many imaging problems. 


Before we investigate maximum likelihood estimation for model selection, let's review some of 
the basic concepts. Let © denote a parameter space (e.g., 9 = R), and assume we have 


observations 
Equation: 


id 


Y; ~ Po (y), 


i=1--,n 


where 0 € Oisa parameter determining the density of the {Y;}. The ML estimator of 0° is 


Equation: 


i Po (Yi 
meres | » (Yi) 


= log Py (Y; 
aoe og Py (Yi) 


= in § > — log Po (Yj). 
argmin ys og Pp (Yi) 


6 maximizes the expected log-likelihood. To see this, let's compare the expected log-likelihood 


of 6° with any other 0 € O. 


Equation: 
Ejlog Pg: (Y)— log Pe (Y)] = 
2 
Why? 
Equation: 
PB * 
E og a (y) | 
Po (y) 


E og 


Py (Y) | 
Py (Y) 
Po (y) 


J log Psu) Po (y)dy 


K(Po, Po) the KL divergence 
0 with equality iff Py» = Pp. 


On the other hand, since 6, maximizes the likelihood over 0 € O, we have 
Equation: 


y log =) log Po: (¥;)— log P; _ (i) <0. 


Therefore, 
Equation: 

1< Po 

— Slog = (¥1) K (P, Py) +K(P Py) 0 

i=l P, ( ) 
or re-arranging 
Equation: 
‘7 Pe (Yi) 
K (P), Pe) <|—>- log —K (P;,, Px) 
= dX Pi (Yi) se 


Notice that the quantity 
Equation: 


is an empirical average whose mean is K(P, Pg«). By the law of large numbers, for each 0 € O, 
Equation: 


a.s. 


— K (Py, Py)| “3 0. 


1< P, 
= dole 
i ee fe 


If this also holds for the sequence {6n, then we have 


Equation: 
1 Py (¥;) 
K (P,Py) < = ie OK (Pi, Pe) + 0asn— co 
On 6 Se ~ PW) On Q 


which implies that 
Equation: 


Py — Py 


which often implies that 
Equation: 


6, > 6 


in some appropriate sense (e.g., point-wise or in norm). 


Example: 
Gaussian Distributions 
Equation: 
1 rv? 
fen Gr ee 
a (y) ve 

Equation: 

O=R, {Yi}. © Py (y) 
Equation: 

Py (y 
K(Pa Pr) = flog EO Py (yay 
Po (y) 
= f |w-9°- (v0) ] Pr wav 
2 
= Eg |(y-6)’] - Ey (v- 6") | 
= Ey (Y* —2Y0+67| -1/2 
* 2 * 
= (6°) +1/2-26°0+ 6-1/2 
2 

sae) 

Equation: 


= 6° maximizes E [log Py (Y)| wrt 6 € O 


Equation: 


bn, = argmax {- ey (Y; — @)} 
argmin {> (Y; - @)°} 
ae 


Hellinger Distance 


The KL divergence is not a distance function. 
Equation: 


K (Po,, Po.) la K (Po,, Po,) 


Therefore, it is often more convenient to work with the Hellinger metric, 


Equation: 
1 
1 1\2 z 
H (Po,, Po,) = (/ (2; — Pt) av) : 


The Hellinger metric is symmetric, non-negative and 
Equation: 


HT (Po,, Po.) H (Po,, Po,) 
and therefore it is a distance measure. Furthermore, the squared Hellinger distance lower bounds 


the KL divergence, so convergence in KL divergence implies convergence of the Hellinger 
distance. 


Proposition 1 
Equation: 


H? (Po,, Po) < K (Po,, Po,) 


Proof: 
Equation: 


(P,P) = | (VP.@- Pa) 4 

[Poway [Po wdv—2 | y/Po, Wy! Po, way 

~ 2-2 if Po, (uy) Po, (y)dy, since / Py (y)dy = 16 
= 2 (1 — Ep, Po, (Y)/Pa, |) 

2 log (zy P00) )/Po, (Y ))). since — 2 < — log « 


IA 


lA 


2 Eo, og Po, (Y)/Po, (Y VP OPW), by Jensen’s inequality 


= Ep,[log (Ps, (Y)/Pa, (Y))] = K (Pa, Po.) 


Note that in the proof we also showed that 


Equation: 
H (Po, Fo.) =2(1~ f y/Pa.(oy/Fo, Way) 


and using the fact log x < x — 1 again, we have 


Equation: 
H(P5,, Po,) < —2 log (/ Pa, (y) 


The quantity inside the log is called the affinity between Pg, and Pg,: 
Equation: 


Po, ( wy) ‘ 


A(Pa..Po,) = f y/Pa )y/Po, (dy. 


This is another measure of closeness between Pg, and Po,. 


Example: 
Gaussian Distributions 
Equation: 


Equation: 


—2 log i | Po, (y) 


2108 f (= 


T 


Po, (y)dy 


1 


VG 


a) os 


(y-01)? | (y-62)? | 
eaeeg | ee AR + 6 Nechemenig arises! 
| 2 2 i) 


Cook 


= -—2 log Hee! 
= =(61 — 9)” 
Equation: 
=> —2 log A(Ps,, Ps,) = 5 (1 _ 02)? for Gaussian distributions 
Si Be oes 5 (61 — 62)" for Gaussian. 


Example: 
Poisson Distributions 
If Po (y) = cas 6 > 0, then 


Equation: 
—2 log A(Py,, Ps,) = (v/a - Von). . 
Equation: 
Summary 
Y; © Py 


1. Maximum likelihood estimator maximizes the empirical average 


Equation: 


1 n 
— J log Py (Yi) 
uel 


(our empirical risk is negative log-likelihood) 
2. 0° maximizes the expectation 
Equation: 


E 


eet 
— log Po (Y; 
pee) 


(the risk is the expected negative log-likelihood) 
3. Equation: 


a.s 


i= : 
— Slog Py (Yi) > E 
ET 


1 n 
— S° log Pp (¥;) 
(ET 


SO we expect some sort of concentration of measure. 
4. In particular, since 
Equation: 


il a Po: (x) a.s. 
— log ——— — K (Po, Pe 

we might expect that K (P; , Py) — 0 for the sequence of estimates {P; Ve ; 
re © 2) = 


So, the point is that maximum likelihood estimator is just a special case of a loss function in 
learning. Due to its special structure, we are naturally led to consider KL divergences, Hellinger 
distances, and Affinities. 


Maximum Likelihood and Complexity Regularization 


Review : Maximum Likelihood Estimation 


In the last lecture, we have n i.i.d observations drawn from an unknown distribution 
Equation: 


iid. ‘ 
Y; oN Po >= {1, Sig di 
Equation: 


where 0° € O. 


With loss function defined as / (0, ¥;) = — log po (Y;), the empirical risk is 
Equation: 


re 12 
R, = -——S~ log po (Yi). 
as og po (Yi) 


Essentially, we want to choose a distribution from the collection of distributions within the parameter space that 
minimizes the empirical risk,i.e., we would like to select 


Equation: 

D5 © P = {Potoco 
where 
Equation: 


— 


0, = i = i Y; * 
arg min 2, og po (Yi) 


The risk is defined as 
Equation: 


R (8) = E[L(6,Y)] = —E log po (¥)]. 


Note that 6” minimizes R(6) over O. 
Equation: 


6° = argmin —E [log po (Y)| 
6cO 


= i — 1/1 Da ; 
argmin / og po (y) - Por (y) dy 


Finally, the excess risk of @ is defined as 
Equation: 


R(A) — R(6") = ic Po) 5. (y) dy = K (po, pg) - 


We recognized that the excess risk corresponding to this loss function is simply the Kullback-Leibler (KL) 
Divergence or Relative Entropy, denoted by K(po,, pe,). It is easy to see that K (pg, , pe,) is always non-negative 
and is zero if and only if pg, = po,. KL divergence measures how different two probability distributions are and 
therefore is natural to measure convergence of the maximum likelihood procedures. However, K (po, , pe, ) is not a 
distance metric because it is not symmetric and does not satisfy the triangle inequality. For this reason, two other 
quantities play a key role in maximum likelihood estimation, namely Hellinger Distance and Affinity. 


The Hellinger distance is defined as 
Equation: 


1 
2 


H (po,,P6,) = (/ (y/0», @) - vm) a) 


We proved that the squared Hellinger distance lower bounds the KL divergence: 
Equation: 


H? (p6,, Po.) < K(po,,P0,) 
H? (po,,Pe.) < K(po,;Pe,) - 


The affinity is defined as 
Equation: 


A (p6,; Pe.) = Po; * Po, (y) dy. 
Iv 


we also proved that 
Equation: 


H? (po,,P9,) < —2 log (A(po, , pe.) - 


Example: 
Gaussian Distribution 
Y is Gaussian with mean @ and variance o?. 


Equation: 
1 (0)? 
Poe (y) = ee 
V2n0? 
First, look at 
Equation: 
Po, 1 2 2 
lo = 6; -—90 2(6, —0 : 
aes [(0; — 03) — 2 (1 — 02)y] 
Then, 


Equation: 


Po, 
K(po,, Pe.) = Eo, og | 


Po, 
6; — 62 2(01 — 42) 
= : d 
oa es / y Po. (y) dy 
ElY]=62 
24 
1 2 2 (9; = 62) 
= 57 (8+ -21m) =“ 


—2 log A(po,,Pe,) 


1 0)? \ 1/2 1 yo)? \ 1/2 
—2 log i ( e€ 2% ) . ( e€ 2%? ) dy 
V27o? V 2102 


see (f gig a) 
= —2 log e 4 do y 
Qra? 


e€ 
V 2ro? 


01-0 \? 
ee aa 
= -2loge ” 2» 
6, — 02)" 
= ee = —=K(po,,po,) > H? (p9,,P9,) - 


Maximum likelihood estimation and Complexity regularization 
Suppose that we have n i.i.d training samples, {X;, Yi}; Bes DPXY . 


Using conditional probability, pyxy can be written as 
Equation: 


PXY (x,y) = Px (x) * PY |X=a (y). 


Let's assume for the moment that px is completely unknown, but py|x—, (y) has a special form: 
Equation: 


PY|X=z (y) = Pri (z) (y) 


where py|x=z (y) is a known parametric density function with parameter f * (a). 


Example: 
Signal-plus-noise observation model 
Equation: 


where W; By, i (0, a”) and X; eae px. 
Equation: 


Y|X =z ~ Poisson(f* (zx)) 
Equation: 


The likelihood loss function is 
Equation: 
U( f(x), y) ct log PxXY (X, x) 
— log px (X)— log py|x (Y|X) 
— log px (X)— log pyx) (Y)- 


The expected loss is 
Equation: 
Ell(f(X),Y)] = Ex[Eyix (l(f(X), Y)|X =a] 
= Ex [ Ey x [- log px (x)— log P H(z) (Y)|x = a| | 
= —Ex[log px (X)] — Ex [ Ey)x [log pye) (Y)|X = 2] | 
= —Ex (log px (X)|—E [log Pf(X) (Y)| . 


Notice that the first term is a constant with respect to f. 
Hence, we define our risk to be 
Equation: 


R(f) = —Ellog pyx (Y)] 
= —-Ex | Eyjx [log pya) (Y)|X = 2] | 


= - / ( ii log pj(2) (Y) - PF*(x) (Y) ay) px (x) dex. 


The function f* minimizes this risk since f (x) = f* (x) minimizes the integrand. 
Our empirical risk is the negative log-likelihood of the training samples: 
Equation: 


n 


= 1 
Rn (f) = — | — log pax, (¥%): 


i=1 


The value - is the empirical probability of observing X = X;. 
Often in function estimation, we have control over where we sample X. Let's assume that 2 = (0, inf and 
Y — R. Suppose we sample 2 uniformly with n = m@ samples for some positive integer m (ie., take m 


evenly spaced samples in each coordinate). 
Let x; i = 1, ...,m denote these sample points, and assume that Y; ~ p;*(z,) (y). Then, our empirical risk is 


Equation: 

— li, li, 
Ra (f) = — DUE (@i), Yi) = — 97 — log pyey (Mi). 
i=1 


i=1 


Note that x; is now a deterministic quantity. 


Our risk is 
Equation: 


n 


RUA) = -— > Blog rye) (%)] 


1 
Pt SS if log P4(x,) (Yi) * Px (x,) (Yi) Ay 


i=1 


The risk is minimized by f *, However, i * is nota unique minimizer. Any f that agrees with f * at the point 
{x;, Y;} also minimizes this risk. 

Now, we will make use of the following vector and shorthand notation. The uppercase Y denotes a random 
variable, while the lowercase y and x denote deterministic quantities. 


Equation: 
Y Yi Sil 
Y2 Yy2 x2 
Y — y = 3 i 
Then, 


ps (Y) = [Tia pif (ws) (random) 

pr(y) = [Ti p(yilf (w:)) (deterministic) . 

With this notation, the empirical risk and the true risk can be written as 
Equation: 


iG) = -= tog v7 (¥). 
Rf) = -—Bllog ps (¥)] 


= -= [108 2; (v) -pp (y) dy. 


Error Bound 


Suppose that we have a pool of candidate functions ¥, and we want to select a function f from F using the 
training data. Our usual approach is to show that the distribution of R,, (f) concentrates about its mean as n 
grows. First, we assign a complexity c(f) > 0 to each f € F so that $+ 2~°/) < 1. Then, apply the union bound 


to get a uniform concentration inequality holding for all models in ¥. Finally, we use this concentration 
inequality to bound the expected risk of our selected model. 


We will essentially accomplish the same result here, but avoid the need for explicit concentration inequalities and 
instead make use of the information-theoretic bounds. 


We would like to select an f € F so that the excess risk is small. 
Equation: 


0 < R(f)-R(F‘) 
= “5 [log pp (Y)— log py (Y)| 
2, Weber > 
= al g p;(Y) 
7 =k (py, P;*) 
where 
Equation: 


is again the KL divergence. 


Unfortunately, as mentioned before, K (p frP r) is not a true distance. So instead we will focus on the expected 


squared Hellinger distance as our measure of performance. We will get a bound on 
Equation: 


“ai? (ps (VY), pp (Y))] = 2y(f (yon (ui) - Pree 9) an). 


Maximum Complexity-Regularized Likelihood Estimation 


Theorem 
Li-Barron 2000, Kolaczyk-Nowak 2002 


Let {x;, Y;}/_, be arandom sample of training data with {Y;} independent, 
Equation: 


Yi~ pee) (Yi) st= Lyn 


, * 
for some unknown function f . 


Suppose we have a collection of candidate functions , and complexities c(f) > 0, f € F, satisfying 
Equation: 


S> 2-elf) <1. 


fEF 


Define the complexity-regularized estimator 
Equation: 


> : i< 2c(f) log 2 
<— eh oe Ye 
wen { 7 ps og py (Vi) + 


n 


Then, 
Equation: 


BL (Pr spp (¥))] < ~—Bllog (A(pr W), Pr (Y)))] 


IA 
B 
=] 
—— 
|e 
x 
Ss 
i 
As) 
ise 
+ 


Before proving the theorem, let's look at a special case. 


Example: 
Gaussian noise 


Suppose Y; = f(ai)+W; ,W; ue Ke (0, a). 


Equation: 
1 z yi- f(x)” 
PHla; Yi) = e 202 . 
ee V 270? 
Using results from example 1, we have 
Equation: 
“Blog Anz (¥),Pp (Y)) = D2 —2l08 A( 7.) Pro (#0) 
= my —2 tos f y/Pr05 (yi) - Pym) (Ys) dys 
lore Zi 
= 47 (ae) -F (=) 
i=1 
Then, 
Equation: 


n 


7 Bll ACen er] = zeag 2 (Feed -F 0) |] 


i=1 


We also have, 
Equation: 


1 
—K (pps) = 


— log p; (Y) 


Combine everything together to get 
Equation: 


202 n 


{2 : aA 2p ies? | 


The theorem tells us that 


Equation: 
(Fn (0) —F° (a). 
1 n (24) — AGH i, <2 (f (2%) -—f° (x;)) : 2c(f) log 2 
ae o? — =D 20? n 
or 
Equation: 


7 P|) F ed) | <my (3 ee) ee | 


Now let's come back to the proof. 


Proof 
Equation: 
2 
#(vp,¢) = [ (yor @- yor) ay 
< -210¢ (| rg @-Pr Wav) 
eS 
affinity 

Equation: 

=> 
Equation: 

1 


B|H?(p7,py)| < 2B} log [pW -Pp Way 


Now, define the theoretical analog of Fra: 
Equation: 


. 1 2c(f) log 2 
n= ES Ct yo eee 
fn ~argmin { nh (PnPr) + 


Since 
Equation: 


= . 2c(f) log 2 
de BRR 247) "s\ 
(og p; (Y) ~ 2e() tog 2) 


(log ps (Y) — 2c (f) log 2) 


“sm 


Pf (Y) . ee 


we can see that 
Equation: 


Pr (Y) : —c (a) log2 


Php (Y)e—elfn )log2 = 


Then can write 
Equation: 


(poe co) ——- + 
B|H (»; Pf )| < 2E}log I \/pz @) py (w) dy 


Pz, We eo ¢(Fn)loe2 7 
< 2E|log 


Py, (Yeo f (PF dy 


¥) 
to get 


Now, simply multiply the argument inside the log by Vz WY) 


Equation: 


| rE Yr te, 
B|H ( 7Pr)| 2 Blog 


VPI Y) s/pp (vr) © | f \/pq() Pp ) ay 
- fie (2682) ] 00 
P75) | oo(fa) oe? 


(Y) Sf y/p7z) Pp Ww) ay 
= K(pj,,p7*) + 2¢ (fn) log 2 


Pz (Y) o¢(Fn) loa 
Pr (Y) J 4/P7 (y) “PF (y) dy 


IA 


+2E | log 


The terms K (p fn» P r) + 2c(fn) log 2 are precisely what we wanted for the upper bound of the theorem. So, to 
finish the proof we only need to show that the last term is non-positive. Applying Jensen's inequality, we get 
Equation: 


——__ ea p7(Y) 
V/ be (Y) oe (Fa) ioe 2 (72)toe2 i/ a (¥) 


<2log | EJe 


Pr (Y) | f [P= @) Pp way J [bp Pp) ay 


2E | log 


Both Y and cm are random, which makes the expectation difficult to compute. However, we can simplify the 
problem using the union bound, which eliminates the dependence on f,,: 


Equation: 
ae ras p7(Y) 
p> (Y) ~e(F, log 3 (Y) 
2E | log v: fj g < 2log | B] S~ eee? . : 
pp (Y) Sf 4/pz(y)- py (y) dy feF J 4/ps (y) - pp (y) dy 
py (¥) 
=| p(Y) 
= 2log 2-4) 
feF S/r (y) pyr (y) dy 
= 2log (= r) 
SEF 
< 0 


where the last two lines come from 


Equation: 
Y 
| a / oar Ef (way = | y/o, (y) - py (y) dy 
and 
Equation: 


Denoising II: Adapting to Unknown Smoothness 


Review: Denoising in Smooth Function Spaces I - Method of Sieves 


Suppose we make noisy measurements of a smooth function: 


Equation: 
Y; = f° (%;) +W;,7= {1, vey HH, 
where 
Equation: 
W; ~ N (0 oO ) 
and 
Equation: 


-() 


The unknown function f “isa map 
Equation: 


f :(0,1) >R. 


In Lecture 4, we consider this problem in the case where f* was Lipschitz on (0, 1]. That is, f * satisfied 
Equation: 


* 


f ()-f (s)|<Llt—s|, Vt,s € [0,1] 


bs L > Ois a constant. In that case, we showed that by using a piecewise constant function on a partition of 


ns equal-size bins [link] we were able to obtain an estimator Fra whose mean square error was 
Equation: 


E|\| ff. (| = 


of‘ 
s 
wo|to 
VS 


1 
Hl O41 u2 os t4 ie 08 uw? 08 og 1 
n> bins 


Example of the piecewise constant approximation of 
* 


f 


In this lecture we will use the Maximum Complexity-Regularized Likelihood Estimation result we derived in 
Lecture 14 to extend our denoising scheme in several important ways. 


To begin with let's consider a broader class of functions. 


Hélder Spaces 


For 0 < a < 1, define the space of functions 
Equation: 


H* (Ca) = {i ore sup Ae ae Ae) Se o.| 


for some constant Cg < oo and where f € L...H® above contains functions that are bounded, but less smooth 
than Lipschitz functions. Indeed, the space of Lipschitz functions can be defined as H! (a = 1) 
Equation: 


H(C,)= {i < Ci op edhe) = He) < a 


[>| 7 


for C, < oo. Functions in H! are continuous, but those in H®%, a < 1, are not in general. 


Let's also consider functions that are smoother than Lipschitz. If a = 1 + 8, where 0 < @ < 1, then define 
Equation: 


H°(C,)= {f 2H*(C,): of e HP (Ca)}. 


In other words, H°®,1 < @ < 2, contains Lipschitz functions that are also differentiable and their derivatives are 
Hdélder smooth with smoothness 6 = a — 1. 


And finally, let 
Equation: 


H? (Cy) = ie a eH} (ca) 


contain functions that have continuous derivatives, but that are not necessarily twice-differentiable. 


If f € H% (C.), 0 < a@ < 2, then we say that f is H6lder—a@ smooth with Hélder constant C',. The notion of 
Holder smoothness can also be extended to a > 2 ina straightforward way. 


Note: If ay < @ then 
Equation: 


fcH’sfeH™. 


Summarizing, we can describe Holder spaces as follows. If f- € H°(C) for some 0 < a < 2 and Cy < 00, 
then 


° ()0<a<l1 |f” (4) -— f (s)| < Calt — s|° 
© (il <a <2 4 O=2@)|2eiea 


Note that in general there is a natural relationship between the Hélder space containing the function and the 
approximation class used to estimate the function. Here we will consider functions which are H6lder—a@ smooth 
where 0 < a < 2 and work with piecewise linear approximations. If we were to consider smoother functions, 
a > 2 we would need consider higher order approximation functions, i.e. quadratic, cubic, etc. 


Denoising Example for Signal-plus-Gaussian Noise Observation Model 


Now let's assume f° € H® (C,) for some unknown a(0 < a@ < 2); i.e. we don't know how smooth f* is. We 
will use our observations 
Equation: 


Y; = a (xi) + Wi, i= {1, gh 


to construct an estimator Fa. Intuitively, the smoother f * is, the better we should be able to estimate it. Can we 
take advantage of extra smoothness in f” if we don't know how smooth it is? The smoother f” is, the more 
averaging we can perform to reduce noise. In other words for smoother f* we should average over larger bins. 
Also, we will need to exploit the extra smoothness in our approximation of f *. To that end, we will consider 
candidate functions that are piecewise linear functions on uniform partitions of [0, 1]. Let 

Equation: 


GF, = lf|<C:  f is piecewise linear on [0,+),{+,~),-..[4%*,1) and the 
coef ficients of each line segment are quantized tox log n bits. 


vn 
levels 


(i-1¥k ik 


Example on the quantization of f on interval — : ) 


The start and end points of each line segment are each one of 4/n discrete values, as indicated in [link]. Since each 
line may start at any of the ./n levels and terminate at any of the 1/n levels, there are a total of n possible lines for 
each segment. 


Given that there are k intervals we have 
Equation: 


| F,| = n* Slog |.F;| = k log n. 


Therefore we can use k log n bits to describe a function f € Fp. 


Let 
Equation: 


Construct a prefix code for every f € F by 
Equation: 


(i) Use 000---1 to encode the smallest k such that f € Fy; 
Bits 
(ii) Use k log n bits to encode which element of 7; we are considering. 


Thus, if f € Fz, then the prefix code associated with f has codeword length 
Equation: 


c(f) =k+k log n = k(1+ log n) 


which satisfies the Kraft Inequality 
Equation: 


S> 2-clf) <1, 


fEF 


Now we will apply our complexity regularization result to select a function f, from ¥ and bound its risk. We are 
assuming Gaussian errors, so 
Equation: 


(vi — f(+))’ 


202 


— log ps (Yi) = + constant. 


We can ignore the constant term and so our empirical selection is 
Equation: 


n —_ a 2 
fa -avepin { 29 iG) - oa? 


= 2c? n 
i=1 


We can compute f,, according to: 


For k= 1, «357% 
Equation: 


,\\2 
1) ep 1 i FG) 
fr Sen R, (f) =argmin 2, 


feFr n 20? 
then select 
Equation: 
a = 2k(1+ 1 log 2 
k =arg min {®. (7) eure es \ 
=1,...,n n 
and finally 
Equation: 


= _ 2(i) 
fn = fn’. 
Because the KL divergence and —2 log affinity simply reduce to squared error in the Gaussian case (Lecture 14), 


we alrive at a relatively simple bound on the mean square error of f,, 
Equation: 


EA(e(s)-))] EECG)-2G0)) ) 


The first term in the brackets above is related to the error incurred by approximating f* by an element of ¥. The 
second term is related to the estimation error involved with the model selection process. 


Let's focus on the approximation error. First, suppose f € H*(Cq ) forl <a <2. Let fr be the “best" 
piecewise linear approximation to f”, with k pieces on intervals [0, r)s [z > : | eee [A - ; 1); Consider the 


; +). By applying Taylor's theorem with remainder 


s * * m a 
difference between f and f, on one such interval, say a ; 


we have 
Equation: 
+ [1 of* z 
t t')(t 
rwo=F(F)+Lo(e-Z) 
fort € [+ +) and some t’ € [z, +]. Define 
Equation: 


£wo=s(F) | (zy ( i) 


Note that fi (t) is not necessarily the best piecewise linear approximation to f * just good enough for our 
purposes. Then using the fact that f° € H*(C.), fort € [i — 1/k,i/k) we have 


Equation: 
®-KO| = Fo 7) -F(z)(: | 


1/df°,, Of (i 
< £0-F£G) 


IA 


So, for all  € [0, 1] 
Equation: 


f(t) f, O| < Cok. 


* * 
Now let f, be the element of ¥;, closest to f;, (f; is the quantized version of f;,) 
Equation: 


IP O-KhO+KO a) 

if O-hO|+|K O-feO| 
1 

Ck * + — 

as 


n 


f(t) — fe (| 


IA 


IA 


since we used + log n bits to quantize the endpoints of each line segment. Consequently, 
Equation: 


PO-KO! +2 O-AKO||A O- FeO] + [fe O— FeO) 
Crp 400, R= 4 2b, 
vain 


A 


IA 


Thus it follows that 
Equation: 


ste {2 ‘ (r ijn) —f" (i/n)) | 807c(f) log 2 < 202k 4 AC. k-* | 2 | 807k (log n + 1) Ic 


fEFx | rn A Jn n n 


The first and last terms dominate the above expression. Therefore, the upper bound is minimized when k~?¢ and 


4 are balanced. This is accomplished by choosing k = nae | . Then it follows that 


7 2 us 1 * 4 a 8a7c(f) log 2 = aaa 
Ut) ee 


If a = 2 then we have 
Equation: 


Equation: 


= O(n* log n). 


lif ¢H °(Cq) for0 < a < 1, let fi be the following piecewise constant approximation to f”. Let 
Equation: 
rt (t) r z ie 1 i-1 i 
= — } on interval | ——, — }. 
n kok 


Then 
Equation: 


fF (i) — fe (0) 


IA 
XQ 
) 


< Cak *. 


Repeating the same reasoning as in the 1 < @ < 2 case, we arrive at 


Equation: 
1 aft eft 
2 ya|(a(=)-s G) 
t=1 


for 0 < a < 1. In particular, for a = 1 we get 


= O(n-er log n) 


Sale) oft 


within a logarithmic factor of the rate we had before (in Lecture 4) for that case! 


Summary 


1. i can be computed by finding least-square line fits to the data on partitions of the form 
[o, +), [4,2),...[44, 1) fork = 1,...,n, and then selecting the best fit by the k that gives the minimum 
of the complexity regularization criterion. 

2. If f° € H*(Cq) for some 0 < a@ < 2, then 


Equation: 
os 1 aft Af VS" 
use(f) = + 9-8|(4()-7(2)) 
1=1 


3. Fn automatically picks the optimal number of bins. Essentially fo (indirectly) estimates the smoothness of f* 


2a 
and produces a rate which is near minimax optimal ! (n «+7 is the best possible). 
4. The larger a is the faster the convergence and the better the denoising ! 


= O(n log n). 


Nonlinear Approximation and Wavelet Analysis 


Review 


In Lecture 4 and 15, we investigated the problem of denoising a smooth 
signal in additive white noise. In Lecture 4, we considered Lipschitz 
functions and showed that by filling constants on a uniform partition of 
width n~'/3 we can achieve an n 2/3 rate of MSE convergence. 


In Lecture 15, we considered Holder-a@ smooth functions, and we 
demonstrated that by automatically selecting partition width and using 
polynomial fits we can obtain a MSE convergence rate of n~2¢/20+1, 
substantially better when a > 1. Also important is the fact that we don't 


need to know the value of a a priori. The estimator f,, is fundamentally 
different than its counterpart in Lecture 4. 


In both cases fn (t) is a linear function (polynomial on constant fit) of the 
data in each interval of the underlying partition. In Lecture 4, the partition 
was independent of the data, and so the overall estimator is a linear function 
of the data . 


However, in Lecture 15 the partition itself was selected based on the data. 


Consequently, Fn (t) is a non-linear function of the data . Linear estimators 
(linear functions of the data) cannot adapt to unknown degrees of 
smoothness. In this lecture, we lay the groundwork for one more important 
extension in the denoising application - spatial adaptivity. That is, we would 
like to construct estimators that not only adapt to unknown degrees of 
global smoothness, but that also adapt to spatially varying degrees of 
smoothness. 


We will focus on the approximation theoretic aspects of the problem in this 
lecture, considering tree-based approximations and wavelet expansions. In 
the next lecture, we will apply these results to the denoising problem, this 
will bring us up to date with the current state-of-the-art in denoising and 
non-parametric estimation. 


Recall that Holder spaces contain smooth functions that are well 
approximated with polynomials or piecewise polynomial functions. Holder 
spaces are quite large and contain many interesting signals. However, 
Holder spaces are still inadequate in many applications. Often, we 
encounter functions that are not smooth everywhere; they contain 
discontinuities, jumps, spikes, etc. Indeed, the "singularities" (or non- 
smooth points) can be the most interesting and informative aspects of the 
functions. 


Example: 
Functions not smooth everywhere. 


Example of functions not smooth everywhere. (a) 1-D Case (b) 2-D 
Case 


spike 
F(t) 


otherwise smooth 


smoothly varyingn intensiy 


except for edges 


Furthermore, functions of interest may possess different degrees of 
smoothness in different regions. 


Example: 
Functions with different degrees of smoothness. 


Example of functions having different degrees of smoothness. (a) 1-D 
Case (b) 2-D Case 


BR a= 1 


NonLinear Approximation via Trees 


Let B® (C,,) denote the set of all functions that are H% (C,,) everywhere 
except on a set of measure zero. To simplify the notation, we won't 
explicitly identify the domain (e.g., [0, 1] or [0, 1]%): that will be clear from 
the context. 


Example: 


Sets of measure zero 


Sets of measure zero. (a) 1-D Case (b) 2-D Case 


F(t) ~ 


2 


a point has measure 
zero in 1-D 


a smooth curve has 
measure zero in 2-D 


Let's consider a 1-D case first. 

Let f € B® (C,) and consider approximating f by a piecewise polynomial 
function on a uniform partition. 

If f is Holder-a@ smooth everywhere, then by using an appropriate partition 
width k~+ and fitting degree [a] polynomials on each interval we have an 

approximation f;, satisfying 

Equation: 


EE) =e lel Soir 


and 
Equation: 


trie O(n s: 


f(t) 


1/2 t 
Smooth curve with a discontinuity. 


However, if there is a discontinuity then for t in the interval containing the 
discontinuity the difference 
Equation: 


If (t)—fr (¢)| 


will not be small. 


Example: 
Suppose f is piecewise Lipschitz and f; ia a piecewise constant. 


Equation: 
If (t)—fr (t)| » A 


where A is a constant equal to average of f on right and left side of 
discontinuity in this interval. 
Equation: 


= llf- filli, =O(k") 


where k? is the width of the interval. Notice this rate is quite slow. 

This problem naturally suggests the following remedy: use very small 
intervals near discontinuities and larger intervals in smooth regions. 
Specifically, suppose we use intervals of width k~2% to contain the 
discontinuities and the intervals of width k~' elsewhere. Then accordingly 
piecewise polynomial approximation fre satisfies 

Equation: 


\If — falln, =O (k °°). 


We can accomplish this need for "adaptive resolution" or "multiresolution" 
using recursive partitions and trees. 


Recursive Dyadic Partitions 


We discussed this idea already in our examination of classification trees. 
Here is the basic idea again, graphically. 


complete RDP corresponding tree 


pruned RDP 


corresponding tree 


Complete and pruned RDP along with their correspnding tree 
structures. 


Consider a function f € B® (C,) that contains no more than m points of 
discontinuity, and is H® (C.) away from these points. 


Lemma 


Consider a complete RDP with n intervals, then there exists an associated 
pruned RDP with O(klogn) intervals, such that an associated piecewise 


degree [a] polynomial approximation (f),,, has a squared approximation 
error of O(min Cc cama he 


Assume n > k > m. Divide [0, 1] into k intervals. If f is smooth on a 
particular interval J, then 
Equation: 


If (t)—fi, (£)| = O (k*) Vt € I. 


In intervals that contain a discontinuity, recursively subdivide into two until 
the discontinuity is contained in an interval of width n~!. This process 
results in at most logon addition subintervals per discontinuity, and the 
squared approximation error is O(k — 2a) on all of them accept the m 
intervals of width n~! containing the discontinuities where the error is 
O(1) at each point. 


Thus, the overall squared D2 norm is 
Equation: 


If — fall, =O (min (7, n)) 


and there are at most k + logan intervals in the partition. Since k>m, we 
can upperbound the number of intervals by 2klogon. 


Note that if the initial complete RDP has n ~ k?* intervals, then the 
squared error is O(k 2°). 


Thus, we only incur a factor of 2alogk additional leafs and achieve the 
same overall approximation error as in the H® (C,,) case. We will see that 
this is a small price to pay in order to handle not only smooth functions, but 
also piecewise smooth functions. 


Wavelet Approximations 
Let f € L? ([0,1]); f f? (t)dt < co. 


A wavelet approximation is a series of the form 
Equation: 


9) 
f= <f,0j% > Vix 


j>0 k=l 


where c, is a constant (co = { ti (t)at), 


Equation: 


1 
<f,0ie >= / f(t) jx (tat 


and the basis functions ~; are orthonormal, oscillatory signals, each with 
an associated scale 2-4 and position k2-/. w;, is called the wavelet at scale 
2-/ and position k2’. 


Example: 
Haar Wavelets 
Equation: 


Die (t) = 2 (Lgrease)2-1(e-1/2))} — 1 eel2-s(e-1/2),274}) 


wir) 


aia 


(k-1)23 t 
_oil2 
Haar Wavelet 
Equation: 
1 
/ dep =U 
0 
Equation: 
1 ka7 
/ vi, (at = | Qdt = 1 
0 7 (k-1)2-3 
Equation: 


1 
i Wee Vian tal (05 = Ok an 


Note: If f is constant on [2-7 (k — 1), 24k], then 


Equation: 


[ Pin =0. 


Suppose f is piecewise constant with at most m discontinuities. Let 
Equation: 


J-1 23 


i = Get = a Oe 


j=0 k=1 


Then, fz has at most mJ non-zero wavelet coefficients; i.e., 

< f, Wj. >= 0 for all but mJ terms, since at most one Haar Wavelet at 
each scale senses each point of discontinuity. Said another way, all but at 
most m of the wavelets at each scale have support over constant regions of 


fz itself will be piecewise constant with discontinuities only possible 
occurring at end points of the intervals [2~” (k — 1), 2~”k]. Therefore, in 


this case 
Equation: 


i alles = O22). 


Daubechies wavelets are the extension of the Haar wavelet idea. Haar 
wavelets have one "vanishing moment": 


Equation: 
1 
oer 
0 


Daubechies wavelets are "smoother" basis functions with extra vanishing 
moments. The Daubechies-NV wavelet has N vanishing moments. 
Equation: 


1 
| Pile — (Usted =O) lo sacg hh = 1b. 
0 


The Daubechies-1 wavelet is just the Haar case. 
If f is a piecewise degree < N polynomial with at most m pieces, then 
using the Daubechies-NV wavelet system. 


Equation: 
) = 
If — fallz, =O (27°); 

and 
Equation: 

OE 

iO) Ses SO a) 
j=0 k=1 


has at most O(mJ) non-zero wavelet coefficients. fz is called the Discrete 
Wavelet Transform (DWT) approximation of f. The key idea is the same 
as we Saw with trees. 


Sampled Data 


We can also use DWT's to analyze and represent discrete, sampled 
functions. Suppose, 
Equation: 


f=([f(/n), f(2/n),..-,f (n/n) 


then we can write i as 
Equation: 


logon—1 23 


f=o+ >) > <f¥,,>¥,, 
=0 k=1 


where 


Equation: 


Pig = [i,m (1), Wj,k (2), nate Wi,k (n)] 


is a discrete time analog of the continuous time wavelets we considered 
before. In particular, 
Equation: 


SRO HG HO le NS 1 
i=1 


for the Daubechies-N discrete wavelets. 
Equation: 


Thus, we also have an analogous approximation result: If f are samples 


from a piecewise degree < N polynomial function with a finite number m 
of discontinuities, then f has O(mJ) non-zero wavelet coefficients. 


Approximating functions with wavelets 


Suppose f € B® (C,,) and has a finite number of discontinuities. Let f, 
denote piecewise degree-N (NV = |[a]) polynomial approximation to f with 
O(k) pieces; a uniform partition into & equal length intervals followed by 
addition splits at the points of discontinuity. 


f(t) 


1/2 t 


extra break pt at discontinuity 


Then 
Equation: 
f(t) fy (t) 2= O (k! — 202) vt € (0, 1 
Equation: 
=> |f (i/n)—f,(i/n)|? =O (k*)i =1,...,7 
Equation: 


2 —2a 
= 1/n||f - fll}, = 0 (*)) 
and i has O(klog2n) non-zero coefficients according to our previous 


analysis. 


Wavelets in 2-D 


Suppose f is a 2-D image that is piecewise polynomial: 


edge 


A pruned RDP of k squares decorated with polyfits gives 
Equation: 


lf — fellz, =O (k*). 


resolution I/k 
sidelength along edge 


Let f= [f (¢/k, j/k); ;, sample range. 
Equation: 


k 
fn (t) = ss f (a/k, 9/RR)1 ste fi—1/hyi/b)alj—1/k,5 /b)} 


ij=l 
then 
Equation: 
2 —1 
If — fallz, =O (& *) 
O(1) error on k of the k? pixels, near zero elsewhere. The DWT of f has 


O(k) non-zero wavelet coefficients. O(27) at scale 
24,4. = 07 1,i0c login. 


Vapnik-Chervonenkis Theory 


Review of Past Lecture 


In our past lectures we considered collections of candidate function ¥ that 
were either finite or enumerable. We then constructed penalties, usually 
codelengths, for each candidate c(f), f € F, such that }) reg 2f) < 1 This 


allowed us to derive uniform concentration inequalities over the entire set F 
using the union bound. However, in many cases the collections .F may be 
uncountably infinite. A simple example is the collection F of a single threshold 
classifier in 1-d having the form 

Equation: 


fi (z) =1e>H 


and their complements 
Equation: 
16 (t) = Lie<s}- 


Thus, ¥ contains an uncountable number of classifiers, and we cannot apply 
the union bound argument in such cases. 


Two Ways to Proceed 


Discretize or Quantize the Collection 


Example: 
To quantize F 
Equation: 


eee ee) — bee ner aer ay 


q is positive, such that Vf, € F, 
Equation: 


[ |t-fils e/a 


if the density of x is bounded by c > 0. q < ee 


Identical Empirical Errors 


Consider the fact that given only n training data, many of the classifiers in such 
a collection may produce identical empirical errors. Also, many f € ¥ will 
produce identical label assignments on the data. We will have at most 2” unique 
labels. 


f is uncountable, its interceptions are countable and bounded by 2”. n intervals 
with 2 classifier per interval. 


The number of distinct labeling assignments that a class Y can produce on a set 
of n points is denoted 
Equation: 


S(F,n) <2" 


The VC dimension is logS(.F, n). Specifically, VC(.F) = k, where k is 
largest integer such that S (.F,k) = 2* Ex. 2n = 2",n = 2, VC(F) = 2. 


Ex. Consider 
Equation: 


F={f: f(t) =1gsyorf (x) = 1pr<r,t € [0, 1} 


Let q be a positive integer and 
Equation: 


Fo=1s iF @)=1les7nort @) =1eayi < (0) 1,.4,0}} 


and, 


Equation: 


lfol= 2 (q+ 1). 


Moreover, for any f € F there exists an f; € A, such that 


Equation: 
i/q 
[|f@- f,(z)|dx < f ldx = 1/g. 
(i-1)/q 


Now suppose we have n training data and suppose i € ¥. We know that in 
general, the minimum empirical risk classifier will converge to the Bayes 
classifier at the rate of n~!/? or slower. Therefore, it is unnecessary to drive the 
approximation error down faster than n—1/2 So, we can restrict our attention of 
F,,-1/. and, provided that the density of x is bound above. We have 

Equation: 


minjeg_,,R(f)— RB ( f") < Cy,min / If” (e) — f (w)|dx < c/n”. 


Vapnik-Chervonenkis theory is based not on explicitly quantizing the collection 
of candidate functions, but rather on recognizing that the richness of ¥ is 
limited in a certain sense by the number of training data. Indeed, given n i.i.d. 
training data, there are at most 2” different binary labelings. Therefore, any 
collection Y may be divided into 2” subsets of classifiers that are "equilvalent" 
with respect to the training data. In many cases a collection may not even be 
capable of producing 2” different labellings. 


Example 


Consider X = (0, 1]. 
Equation: 


F ={f: f(x) =1pe>yorf (2) = lect € (0, 1]} 


Suppose we have n training data: (x1,...,2,) € [0,1]. With 2° denotes the 
location of each training point in [0,1]. Associated with each x is a label 

y € {0,1}. Any classifier in FY will label all points to the left of a number 

t € [0,1] as "1" or "0", and points to the right as "0" or "1", respectively. For 

t € [0, x1), all points are either labelled "0" or "1". For t € (a1, 22), 21 is 
labelled "0" or "1" and x9...z,, are label "1" or "0" and so on. We see that there 
are exactly 2n different labellings; far less than 2”! 


The number of different labellings that a class Y can produce on a set of n 
training data is a measure of the "effective size" of YF. The Vapnik- 
Chervonenkis (VC) dimension of ¥ is proportional to the log of the effective 
size. Let V(.F, n) denote the VC dimension of .F, typically a constant, 
independent of n. The VC inequality states that for all f € F 

Equation: 


P(|R, (f) — R(f)| > c) < BEV (Fi) e—ne?/32 


This type of uniform concentration inequality can be used in a similar fashion to 
our use of Hoeffding's inequality plus union bound. 


Hyperplane Classifiers 


We will go into the details of VC Theory next lecture, and the remainder of this 
lecture will introduce the key ideas with an example Consider the following 


setup. Let X = (0, 1]*, Y = {0, 1} Let 
Equation: 


F= {f : f (z) = Lieusot | 
with wo and we R¢*! This is the collection of all hyperplane classifiers. F is 
infinite and uncountable. 


Suppose that we have n training data 
Equation: 


{XG Va) 4 


There are at most 27) unique classifiers in Y with respect to these data. To 


see this, consider d arbitrary data points 71, ..., ;,, and let wie + wo > Obea 
hyperplane containing these points. To be specific, take the hyperplane with 
Equation: 


||wow || = 1. 

this hyperplane coincides with two possible classification rules: 
Equation: 

fie) = Lire cies} 
Equation: 

fo (@) = Lywretwy<o} 
Each d-tuple of training data produces two distinct classifiers, assuming the 
data are not co-linear. Thus, there are at most 2* Cs) unique classifiers in F 
with respect to the training data. (All other f € ¥Y produce the same labels and 


empirical risk as one of the classifiers.) Let's enumerate the unique hyperplane 
classifiers fy, ..., Fox (nr), and let 


Equation: 
fi =arg min R,, (f) 
fe {fir-fo(a)} 

and let 

Equation: 

R =infregR(f) 

and define 
Equation: 


f° =argmingegR (f) 


If multiple f € F achieve R’, pick f” to be one of them in an arbitrary fixed 
number. 
Theorem 


Assume that P; has a density, but that the distribution of (a, y) is other 
arbitrary. If n > d and 2d/n < € < 1 then 
Equation: 


Note: The assumption that P, has a density insures that no d+1 points are co- 
planar. This in turn, guarantees that there are exactly 25) unique classifier 


and that the 2 () under consideration are fully representative of all possible 
classifiers in Y, with respect to the data. 


The proof is a specialization of the basic ingredients of VC Theory to the case 
at hand. Here we follow the proof in DGL '96. First we note that, 
Equation: 


n (fs) (1) =" (he) ~Be (Bs) +e (7) (1) 


< R (fn) —R, (Fn) +R, f*) -R(f') 4+d/n 


and since R,, (Ga < R,(f) +d/n for any fe F 


Equation: 


< maz,.1,...9(")(R (fi) — (BR), (fi) + (BR), (f°) — R(f*) + 4/n. 


Therefore, by the union bound: 
Equation: 


We can bound the second term of the above bound using Chernoff's/Hoeffding's 


inequality: 
Equation: 
P(Rn (”) —R (") > €/2— d/n) 
Equation: 
- e72n(€/2—d/n)” 
Equation: 


ae 
Pe ert ne [2 


Next, let's bound one of the terms in the summation. For example, take 
Equation: 


P(R(fi) — Bn (fi) > (€/2)). 


Note that by symmetry all 2) terms will have identical bounds. Since the 
bounds are independent of P,y. 


Assume that f; is determined by the first d data points 714, ..., xg. By the 
smoothing property of expectations we can write, 
Equation: 


P(R (f:) — Bn (f) > </2) 2 B|P(R (f:) — Bn (fi) > €/2\e1, --- a). 


From here, we will bound the conditional probability inside the expectation. Let 
(X1!,Y1"), ... (47, YJ’) be d additional random samples that are independent 
and identically distributed as the data (X1, Y,),...,(Xq, Ya). {X7, Y,” es are 
often called the "ghost sample" since they are not actually observed. They are a 
fictious sample leads to a simple bound on the conditional probability. Define if 
ae 

Equation: 


(x:,¥%)) = (X1,¥/") 
orifi >d 
Equation: 


(x;, Y;) = (Xi, Y;). 


p eoieed 
That is, {X re é |e agrees with our observed data on i>d, but the first d 


samples are replaced with the ghost sample. Then, 
Equation: 


P(R (fi) — Bn (fi) > €/2|z1, nite) 


Equation: 


ae P(r (fi) —1/n S° Lp(a) Ay > €/2\r1, ote] 


i=d+1 


Equation: 


a P(r (fi) = 1/nS— 1 (0) £4; =f d/n o €/2|x1, a4) 
1 


Equation: 


A 


= P(R(fi) = (2), (f:) > t/2 —d/nlzn, insta) 


where, 
Equation: 


R, (ft) = ny mEACAE AG 


Aa 


Note that n (2), (f1) is binomially distributed with mean R(f,) and it is 


independent of x1, ..., gq Therefore, 


Equation: 
P(R(fi) ~ BR (f,) > €/2 —d/nlan, 5a) 
Equation: 
= P(R(fi) — Ri, (fi) > t/2-d/n|ar, ied) 
Equation: 
< e72n(€/2—-d/n)” 
Equation: 


2 
< e2dep—ne [2 


In conclusion, 
Equation: 


P(R (fa) - RS c) 


Equation: 
<5 (RD Rn (ti) > €/2) + P(Rr (f°) —R(f*) +4/n > /2) 
Equation: 
< a ene SC ea 
Equation: 


gs (2("") + Lee, 


Lastly, Corollary If n > d, then 
Equation: 


B|R (Fn) —minjegR ()| < 4/2(d + 1)(logn + 2)/n. 


The Vapnik-Chervonenkis Inequality 


The Vapnik-Chervonenkis Inequality 


The VC inequality is a powerful generalization of the bounds we obtained for the 
hyperplane classifier in the previous lecture. The basic idea of the proof is quite 
similar. Before starting the inequality, we need to introduce the concept of shatter 
coefficients and VC dimension . 


Shatter Coefficients 


Let & be a collection of subsets of #%, definition : The n“” shatter coefficient of & 
is defined by 
Equation: 


ma 


San) = pal {tus tn} (A Aco}. 


L1 5-25 hn 


The shatter coefficients are a measure of the richness of the collection &. Av (n) is 
the largest number of different subsets of a set of n points that can be generated by 
intersecting the set with elements of ./. 


Example: 

In 1-d, Let & = {(—o0, t],te #} Possible subsets of {x1,...,2,} generated by 
intersecting with sets of the form (—oo, ¢] are 

fii Ln Aes ae en ito es Aes Oe ence, (1) — 1, 


Example: 

In 2-d, Let & = { all rectangles in 7} 

Consider a set {r1, Lo. 03, xa} of training points. If we arrange the four points into 
the corner of a diamond shape. It's easy to see that we can find a rectangle in Z? to 
cover any subsets of the four points as the above picture, i.e. Av (4) = 2. 6, 
Clearly, Av (n) = 2",n = 1, 2, 3 as well. 

However, for n = 5, Wg (n) < 2°. This is because we can always select four 
points such that the rectangle, which just contains four of them, contains the other 


point. Consequently, we cannot find a rectangle classifier which contains the four 
outer points and does not contain the inner point as shown above. 

Note the Ay < 2”. 

If |{{21,...,¢@n}() A, Ae @}| = 2” then we say that & shatters x1, ..., Ln. 


VC Dimension 


The VC dimension 
Vy of a collection of sets .& is defined as the largest interger n such that 
Sv (n) — Pe 


Example: 
A = {(—00,t];t€2}, Ay =n+ Lhence Vy = 1. 


Example: 

@ = { all rectangles in #7}. 

Aig th Nd, band of 2 on = 4 Hence: V 7 — 4. 

The VC dimension provides a useful bound on the growth of the shatter coefficients. 


Sauer's Lemma: 


Let & be a collection of set with VC dimension V,yv < oo. Then 
Vn, Ze (n) < 2%, ("). also Sa (n) < (n+1)"",Vn. 
1 


VC Dimension and Classifiers 


Let F be a collection of classifiers of the form f : #4 —> {0, 1} Define 

w= Lies fey 1)-x {0} Lia: fae) — 0} x 41}, fer} In words; this is 
collection of subsets of 2 x Y for which on fe maps the features z to a label 
opposite of y. The size of .& expresses the richness of Y. The larger ./ is the more 
likely it is that there exists an feF for which R(f) = P(f(X) 4 Y) is close to the 
Bayes risk R” = P 6; AOE Y) where f * is the Bayes classifier. The n” shatter 


coefficient of F is defined as Az (n) = Y~v (n) and the VC dimesion of F is 
defined as Vz = Vy. 


Example: 

linear (hyperplane) classifiers in B74 

Consider d = 2. Let n be the number of training points, it is easy to see that when 

n = 1, let & be as above. By using linear classifiers in Z?, it is easy to see that we 
can assign 1 to all possible subsets { {x1}, y} and 0 to their complements. Hence 
ig (AyD. 

When n = 2, we can also assign 1 to all possible subsets 

{{x1,x2}, {x1}, {x2}, y} and 0 to their complements, and vice versa. Hence 
Oe CA Re ted 

When n = 3, we can arrange arrange the point 71, %2, ©3(non-colinear) so that the 
set of linear classifiers shatters the three points, hence Wg (3) = 8 = 2° 

When n = 4, no matter where the points x1, 22, £3, £4 and what designated binary 
values y1, Y2, 3, y4 are. It's clear that 2 does not shatter the four points. To see 
the claim, first observe that the four points will form a 4-gon (if the four points are 
co-linear, or if the three points are co-linear then clearly linear classifiers cannot 
shatter the points). The two points that belong to the same diagonal lines form 2 
groups and no linear classifier can assign different values to the 2 groups. Hence 
egred Ae e aN arial We 

We state here without proving it that in general the class of linear classifiers in Z¢ 
has Vz =d+1. 


The VC Inequality 


Let Xj, ,..., Xp be iid. Z%-valued random variables. Denote the common 
distribution of X;,1 <i <n by uw (A) = P(X,€ A) for any subset A c #2. 
Similarly, define the empirical distribution pp (A) = = 07 1¢x,<4}- 
Theorem 

VC '71 


For any probablilty measure yz and collection of subsets 2, and for any € > 0. 
Equation: 


sup ne 
_ Z ne? /32 
P( ; [ln (A) — uw (A)] > c) < 8S 4 (n)e 


and 
Equation: 


2 ne (A)—4(4)|] s2y EO) 


E 


Before giving a proof to the theorem. We present a Corollary. 
Corollary 


Let F be a collection of classifiers of the formf : #4 — {0, 1} with VC dimension 


Vz < oo, Let R(f) = P(f(X) AY) and Rn (f) = + DT 1ryexyev3, where 
X;,Y;,1 <2 < narei.i.d. with joint distributionPyy. 


Define 


~  argmin~ 


Then 
Equation: 


z[R(f.)| - PERS a 


Let f = {{x: f(x) = 1} x {0} Ute : f(x) = 0} x {1}, fe F} 


Note that 
Equation: 


P(f(X) # Y) = P((X,Y) A) := w(A) 


where A= f(a) 1) & {0} ee Fe) = 0} eT. 


Similarly, 
Equation: 


1< fs er 
TS ives] > teen 
1 1 


Therefore, according to the VC theorem. 


Equation: 
Bl PP Ia) — RN] =2[ oP aA) — a (a)|] < ay) 982% 
1/28 2.72 (n) 
Since Vz < 00, Yz(n) < (n +1)” and 
Equation: 
Bl fe BR, (f)- Rip 25 Vz log es 1)+ log 2 . 

Next, note that 

Equation: 

a(n) (ain = (a(R) -R(A)] + a(R) "hav 
~ [a(7) -R.(R)] + [P%(R. (A) - R00) 
< [a(h) -(F)] +B (Rin - R00) 


< 20M [Ra (f)- RUA) 


Therefore, 
Equation: 


I 


n+ 1)+ log 2 


ny ee 


n 


Applications of VC Bound 


Linear Classifiers 


Suppose ¥= {linear classifiers in R%}, then we have 
Equation: 


Vzg=d+1, es =argmin R,, (f) 
SEF 


Equation: 


(d+ 1) log (n+ 1)-+ log 2 
fCF 7 n 


Generalized Linear Classifiers 


Normally, we have a feature vector X € R%. A hyperplane in R4 provides a 
linear classifier in R“%. Nonlinear classifiers can be obtained by a straightforward 
generalization. 


Let 91,°°+, Pq's d > dbeacollection of functions mapping R? > R. These 
functions, applied to a feature X € R4, produce a generalized set of features, 
vy = (v1 (X), v2 (X),-+-, va (X))’. For example, if X = (1, x2)’, then we 
could consider d’ = § and y = (a1, £2,222, ee 22)’ € R°. We can then 
construct a linear classifier in the higher dimensional generalized feature space 
R!. 

The VC bounds immediately extend to this case, and we have for ¥' = { 


generalized linear classifiers based on maps y : R? > R? }, 
Equation: 


e[R(f)]- ag Rin caf Sete Dee 


Half-Space Classifiers 


Theorem 
Steele '75, Dudley '78 


Let Ybe a finite-dimensional vector space of real-valued functions on R?. The 
class of sets Y = {{x: g(x) > 0} : g € FY} has VC dimension > dim(¥). 


It is sufficient to show that no set of n = dim(¥Y) + 1 points can be shattered by 
so. Take any n points and for each g € Y, define the vector 


Vy = (g (ei). ue G9 (Gnd): 


The set {V, : g € GY} is a linear subspace of R” of dimension < dim (Y) =n — 1. 
Therefore, there exists a non-zero vector @ = (Q@1,-°--,Q@n) € R” such that 

yo, ag (x;) = 0. We can assume that at least one of these a is negative (if all 
are positive, just negate the sum). We can then re-arrange this expression as 
Se, ig (x;) om ae, aig Cae 


Now suppose that there exists a g € Y such that the set {x : g(x) > 0} selects 
precisely the ae on the left-hand side above. Then all terms on the left are non- 
negative and all the terms on the right are non-positive. Since a is non-zero, this is 
a contradiction. Therefore, x1, ---, 2, cannot be shattered by sets in 

{xz : g(x) > 0},g EY. 6.375pt0.0pt6.375pt 


Example: 

Consider half-spaces in R4 of the form 

a = {x ER!:2,>b,i€ {1,:--,d},beE R}. Each half-space can be 
described by 

Equation: 


Ld 
Equation: 


= dim(Y)=d+1, Va <d+1. 


Tree Classifiers 


Let 
Equation: 


R= {recursive rectangular partitions of R? with k+1 cells} 


Let T © %. Each cell of T results from splitting a rectangular region into two 
smaller rectangles parallel to one of the coordinate axes. 


Example: 

TE TEMG Eins ye 

Each additional split is analogous to a half-space set. Therefore, each additional 
split can potentially shatter d + 1 points. This implies that 

Equation: 


Vaz, < (d+ 1)k. 


Example: 

eel 

k = 1 split shatters two points. 

k = 2 splits shatters three points < 4. 


VC Bound for Tree Classifiers 
Equation: 

Fr, = {tree classifiers with k+1 leafs on R*} 
Equation: 


e[R(f)]- ag Rin cay er et 


Exercise: 


Problem: 


How can we decide what dimension to choose for a generalized linear 
classifier? 


How many leafs should be used for a classification tree? 
Solution: 


Complexity Regularization using VC bounds! 


Structural Risk Minimization (SRM) 


SRM is simply complexity regularization using VC type bounds in place of 
Chernoff's bound or other concentration inequalities. 


The basic idea is to consider a sequence of sets of classifiers 1, F2,..., of 
increasing VC dimensions Vz, < Vz, < .... Then for each k = 1, 2,... we find 
the minimum empirical risk classifier 

Equation: 


Atk) he 
n = Ry 
f argmin (f) 


and then select the final classifier according to 
Equation: 


k =argmin « R 
k>1 n 


Bs ie eX 32V z, (1 1 
(i) +4] g, (log n + 1) 


a atk 
and f, = A ) is the final choice. 


The basic rational is that we know 
Equation: 


where C’” is a constant. 


The end result is that 
Equation: 


E|R(fa)| min {i R(f) + 19 “ents 


analogous to our pervious complexity regularization results, except that 
codelengths are replaced by VC dimensions. 


In order to prove the result we use the VC probability concentration bound and 
assume that A = $°,., Vz, < oo. This enables a union bounding argument and 


leads to a risk bound of the form given above. 


Key Point of VC Theory 
Complexity of classes depends on richness (shattering capability) relative to a set 


of n arbitrary points. This allows us to effectively “quantize” collections of 
functions in a slightly data-dependent manner. 


Application to Trees 


Let 
Equation: 


F,={k leaf decision trees in R“}, Vz, < (d+1)(k+1) 
Equation: 
(k) _ 3 
n= Ry, 
fx” =argmin Rr (f) 


Equation: 


—~ 


32(d+1)(k—-1)(0 1 
k =argmin (wn Rin + Hey een 


nr 


Then 
Equation: 


7-7 


n 


satisfies 
Equation: 


E [R (fn) <min (sn R(f)+ joy SNE Diese) 


kz1 \ fe an 


compare with 
Equation: 


a 3k — 1) log 2+ 4 logn 
E|R(f,)| <min min nip+-y Sens Hen 
k>1 fEedyadic k leaf trees 2n 


from Lecture 11. 


Lower Performance Bounds for Estimators 


Lower Performance Bounds 


In other modules, estimators/predictors are analyzed, in order to obtain upper bounds on their 
performance. These bounds are of the form: 
Equation: 


min E E (Gas f)| <Cn 7 


SEF 


where > 0. We would like to know if these bounds are tight, in the sense that there is no other 
estimator that is significantly better. To answer this, we need lower bounds like 
Equation: 


infsup E E (fn; f)| em | lel 


fn £6F 


We assume we have the following ingredients: 


e *Class of models, ¥ C SY. F is a class of models containing the “true” model and is a subset of 
some bigger class .Y. E.g. F could be the class of Lipschitz density functions or distributions 
Pxy satisfying the box-counting condition. 

¢ *An observation model, 7, indexed by f € ¥%. #; denotes the distribution of the data under 
model f. E.g. in regression and classification, this is the distribution of 
Z = (X1,¥1,-++,Xn, Yn) C &. We will assume that Y + is a probability measure on the 
measurable space (2, #). 


¢ *A performance metric d(.,.). > 0. If you have a model estimate fai then the performance of 
that model estimate relative to the true model f is d 2 f) . Eg. 


Equation: 
Regression: d (fas) =l\fa—file=(f (Fale) -1(@)) ar] ° 
Equation: 
Classification: d Ge f) St (G,) —_R = a es |2n (x) — 1|dPx (x) 


As before, we are interested in the risk of a learning rule, in particular the maximal risk given as: 
Equation: 


sup By [d (fas f)] =sup fd (Fa(Z),f)ay (2) 


fEF 


where f,, is a function of the observations Z and Er denotes the expectation with respect to Pf. 


The main goal is to get results of the form 
Equation: 


x A “a 
&,, =infsup E E c2 f)| > CS8n, 
fn SEF 


where c > 0 and s, — 0 as n — ov. The inf is taken over all estimators, i.e. all measurable functions 


~ 


fri Zor. 


Suppose we have shown that 
Equation: 


lim inf s, lg >c>0 (A lower bound) 
n—->oco 


and also that for a particular estimator f n 
Equation: 


lim sup ie sup Ey E (Fa, F)| <C 


n—-oo fEF 


Equation: 


* 
>limsup s,1%, <C, 


n—- oo 


We say that s,, is the optimal rate of convergence for this problem and that f , attains that rate. 
Note:Two rates of convergence W,, and W are equivalent, i.e. V,, = W) iff 


Equation: 


WV WV 
0 <lim inf — <limsup — < o 
N00 Vv, noo ¥y 


General Reduction Scheme 


Instead of directly bounding the expected performance, we are going to prove stronger probability 
bounds of the form 
Equation: 


These bounds can be readily converted to expected performance bounds using Markov's inequality: 


Equation: 
aa) 


Sn, 


P;(d(fuf) > sn) < 


Therefore it follows: 
Equation: 


ree Kf E ia f)| ay Sal? 7 (a (in; f) > sn) > C8y 


First Reduction Step 


Reduce the original problem to an easier one by replacing the larger class Y with a smaller finite class 
{fo,---, fu} C F. Observe that 
Equation: 


pee a coe) 


The key idea is to choose a finite collection of models such that the resulting problem is as hard as the 
original, otherwise the lower bound will not be tight. 


Second Reduction Step 


Next, we reduce the problem to a hypotheses test. Ideally, we would like to have something like 
Equation: 


ae Py (a es f) > sn) Se a Py, (hi, (Z)# i) 


The inf is over all measurable test functions 
Equation: 


hn: £ > {0,---,M} 


and FP ¢, (fn (24 i) denotes the probability that after observing the data, the test infers the wrong 
hypothesis. 


This might not always be true or easy to show, but in certain scenarios it can be done. Suppose d(., . ) 
is a semi-distance, i.e. it satisfies 


* (id(f, 9) = d(g, f) = 0 (Symmetric) 
° (ii) 
Equation: 
a(f, f) =0 
© (iii)d(f,g) < d(h, f) + d(h, g) (Triangle inequality) 
A 
E.g. with f,g: R¢ > R,d(f,9) = ||f —gllo. 


Lemma 


Suppose d(.,. ) is a semi-distance. Also suppose that we have constructed fo,---, fi S.t. 
d( fj, fr) > 28n, Vj # k. Take any estimator is and define the test: 7" o fi: : & — {0,---,M} as 
Equation: 


v* (f,) =aremin d (fa, fi) 


Then Y (Fn) # j, implies d ar fi) > Sn. 


Suppose (Fn) fF eh S49 dk cs fr) <d ce fi). Now 


Equation: 
28n <d(fj, fe) <d cz fi) +d (Fn; fe) < 2d igs fi) 


Equation: 


The previous lemma implies that 
Equation: 


2 (a fats) 00) = (0 (He) #3) 


Therefore, 
Equation: 


IV 


unteup PF; (4 (Fn; fi) 2 sn) 


f, feF inf max Py, (a Ca > sn) 


Ff, Fel foy fu} 


IV 


a ican ee (¥" (7) - i) 


IV 


inf max ‘ ade (fn - j) 


Ry, Je{0,--,M 


as 


Pe. 


The third step follows since we are replacing the class of tests defined by y (in) by a larger class of 


ALL possible tests Rns and hence the inf taken over the larger class is smaller. 
Now our goal throughout is going to be to find lower bounds for Pe,yy. 


So we need to construct fo,---, far s.t.d(f;, fr) > 28n, 7 A k and P..w > c > 0. Observe that this 
requires careful construction since the first condition necessitates that f; and f; are far from each other, 
while the second condition requires that f; and f;, are close enough so that it is harder to distinguish 
them based on a given sample of data, and hence the probability of error P.,14 is bounded away from 0. 


We now try to lower bound the probability of error Pe. We first consider the case M = 1, 
corresponding to binary hypothesis testing. 


M = 1: Let Po and P, denote the two probability measures, i.e. distributions of the data under models 0 
and 1. Clearly if Pp and P, are very “close", then it is hard to distinguish the two hypotheses, and so 
P..1 is large. 


A natural measure between probability measures is the total variation , defined as: 
Equation: 


V (Po, Pi) up Py (A) — Pi (A )|=sup | [ m(2)- (Z)dv (Z) 


where po and pj are the densities of Py and P; with respect to a common dominating measure v and A 
is any subset of the domain. We will lower bound the probability of error P..; using the total variation 
distance. But first, we establish the following lemma. 

Lemma 

Scheffe's lemma 

Equation: 


V(Po, Pi) = 5 | pol) - Z)|dv (Z == | m-pil 
= 1— f min (er) 


Recall the definition of the total variation distance: 
Equation: 


V (Po, Pi) =sup / po — Pi 
A \|JA 


Observe that the set A maximizing the right hand side is given by either {Z © & : po (Z) > pi (Z)} 
or{Z © &: pi (Z) = po (Z)}.- 


Let us pick Ap = {Z € & : po (Z) > pi (Z)}. Then 


Equation: 
1 
V(P,Pi)= | po-m=- |] po-m=— | |po—pil 
Ao Ag 2 


For the second part, notice that 
Equation: 


po (Z)— min (po (4), pi (Z)) 


Now consider 
Equation: 


1~ f min (po,p1) = f po(Z)~ min (po (Z),01 (Z)) = [po (2) ~ ps (Z)av (2) = V (Po, Pa) 


0 


We are now ready to tackle the lower bound on P,.1. In this case, we consider all tests 


hn (Z) : ¥ — {0,1}. Equivalently, we can define hy (Z) = 14 (Z), where A is any subset of the 
domain. 
Equation: 


IV 


P., =inf max &Y; (fn ) 
Rn JE{0,- + -,M} 7 d 


inf (FP (in 4 0) +P; (in sf i)) 


inf Py (La (Z) #0) + Pi (La (Z) #1) 


So if Po is close to P;, then V(Ppo, P) is small and the probability of error P-,; is large. 


This is interesting, but unfortunately, it is hard to work with total variation, especially for multivariate 
distributions. Bounds involving the Kullback-Leibler divergence are much more convenient. 


Equation: 


= fog 425, (ray (Z) = f tog BA 
K(P.|\Ps) =f rN aaa (Z)= f ee 


The following Lemma relates total variation, affinity and KL divergence. 
Lemma 


1—V (Po, Pi) => 3A? (Po, Pi) = > exp (-K(P1 ||Po)) 


For the first inequality, 


Equation: 
(fore) 
(/ ymin (po, pi) max (on.r1)) 


(/ min (po, p1)/max (on.r1)) 


/ min (po, p1) : max (po, P1) by Cauchy-Schwarz inequality 


= / min (po, P1) (2 = / min (v.21) ) * fmin(po,p1)+ fmax(po,p1)=f po+f pi=2 


2 f min (a) 


2(1 — V (Po, Pi)) 


A? (Po, P:) 


IA 


IA 


For the second inequality, 
Equation: 


wisn) ie) 


= csr (ioe ( 

= exp (2 log ne 

= exp (2 log (f /2)) 
Pi 

> exp (2 / log ( |  ) vs) by Jensen’s inequality 
Pi 


Putting everything together, we now have the following Theorem: 

Theorem 

Let F be a class of models, and suppose we have observations Z distributed according to Ys, f € F. 
Let d ions f) be the performance measure of the estimator fh, (Z) relative to the true model f. Assume 


also d(.,.) is a semi-distance. Let fp, f; € F bes.t.d(fo, f,) > 28,. Then 
Equation: 


5 
=e 
wn 
i= 
ue) 
S 
“=~ 
Qu 
——™~ 
=”) 
, 
VS 
V 
w 
3 
aa 
IV 


inf ws gs (a (iat) Sa sn) 


fa je {0,1} 


IV 


1 
zoe (—K (Py, ||Py, )) 


How do we use this theorem? 


Choose fo, fi such that K(P,||Po) < a, then P.1 is bounded away from 0 and we get a bound 
Equation: 


ae Py (a (fa, f) > sn) >c>0 


or, after Markov's 
Equation: 


infsup Es E (Fn: f)| > CSy 


fan FEF 


To apply the theorem, we need to design fo, fi s.t.d (fo, fi) > 2s, and exp (—K(Py, ||Py, )) > 0. 
To reiterate, the design of fo, f1 requires careful construction so as to balance the tradeoff between the 
first condition which requires fo, f; to be far apart, and the second condition which requires fo, f; to 
be close to each other. 


Example: 

Lets use this theorem in a problem we are familiar with. Let X € [0,1] and 

Y|X = z ~ Bernoulli(n(z)), where n(x) = P(Y = 1|X =z). 

Suppose G = e 1] . We proved that under these assumptions and an upper bound on the density of 
X, the Chernoff bounding technique yielded an expected error rate for ERM 


Equation: 
= [2(@,) -x]-0( /***) 
n 
Is this the best possible rate? 


Construct two models in the above class (denote it by Y), Po) and PY). For both take 
Px ~ Uniform ((0, 1]) and 79) = 1/2 — a, nq) = 1/2 + a(a > 0), so Gy = 0, G, = [0, 1]. 
We are interested in controlling the excess risk 


Equation: 
R (Gn) =i (c") = i |2n (2) — 1]dPx (2) 
G, AG* 
Note that if the true underlying model is either PY) or P&, we have: 
Equation: 
R; (Gn) =, (G;) = i. _ (2; (@) — 1|\dz = 2a [ _dxz = 2ad, (Gn,G;) 
G,AG; G,AG; 


Proposition 1 
da (.,.) is a semi-distance. 


It suffices to show that d (G1, G2) = d(G»2,G,) > 0, d(G, G) = OVG and 
d(G1, G2) < d(Gi, G3) + d(G3, G2). The first two statements are obvious. The last one (triangle 
inequality) follows from the fact that G; AG2 C (Gi; AG3) U (G3AG_). 


Suppose this was not the case, then da: x € Gj AG s.t.x € Gi: AG3 and x ¢ G2AG:3. In other 
words, 
Equation: 


rE (Gi AG.) NM (G,AG3)° M (G2AG3)° 


Since SAT = (SN T°) U(S° NT), we have: 


Equation: 
zg € [(G1NG5)U (GUN G2)] N [(G{ U G3) N (G1 U G3)] N [(G5 U G3) N (G2 U G3)| 
€ [G,N (Gi UG3)N GSN (G2 U G3)] U [GN (Gi U G5) N G2 N (G5 U G3)] 
€ IG1N G3N G2N G3] U [G{NG3NG2N G3] 
€ @, a contradiction 


Lets look at the first reduction step: 


Equation: 
ee P (R (G,) —R (c") = sn) > aan P (R, (G,) = R; (c;) > sn) 
= a eae P; (as (Gn, G;) > $n/2a) 


So we can work out a bound on dy and then translate it to excess risk. 


Lets apply Theorem 1. Note that d, (G4; G;) = = 1 and let Po = = PO, x,y, and 
A 


2 pit) 
Pi _ PR Vig Xni¥o" 
Equation: 


”) iXnV, (Xi, ¥i,-°+, Xn; Yn) 


PX Y;,--+XnVo 
K(Pi||Po) = Ea, |log = 
PRY Xn¥n (Xi, Yi, anes Xs ¥,) 
: 1 
Py, The ; re . no Yn) 


( ) 
0 0 
Pry, (Xi, Yi) _ ‘Dy ye (Xn; Yn) 
) 


n : Px, y, (Xi, Yj) 

i=1 PX.Y,; (X3, Y;) 
ty [oe Prix (YilX1) 
= 7 ey a 
Py (YiLX1) 


Now Py )y (Y, = 1|X,) =1/2+aand ae (Y, = 1|X,) = 1/2 — a. Also under model 1, 
Y, ~ Bernoulli (1/2 + a). So we get: 


Equation: 


n fara) log an 
= n(2a log (1/2 + a) — 2a log (1/2 — a)| 
1/2+a 
1/2-—a 
1/2+a 

< 2na( ee — i) 
. 

1/2-a 


1/2-a 


K(P||Po) Tata 


+ (1/2 — a) log 


= 2na log 


Leta = 1/./n and n > 16, then K(P, ||Po) < 4n 16. 


1 1 
n 1/2-1//n S 


Using Theorem 1, since da (G3; a) = 1, we get: 
Equation: 


infmax P; (da (G,.G;) >1/2) > 76" 
Gr J 


AS ae 


Taking s, = 1/,/n, this implies 
Equation: 


or, after Markov's inequality 
Equation: 


=~ * 1 1 

infsup E|R (G,) - R(G")] > 7e 6 — 
G, peF? 4 vn 

Therefore, apart from the log n factor, ERM is getting the best possible performance. 


Reducing the initial problem to a binary hypothesis testing does not always work. Sometimes we need 
M hypotheses, with IM — oo as n — oo. If this is the case, we have the following theorem: 


Theorem 2 Let M > 2. {fo,---, fur} © F be such that 


¢ .d(f;, fz) > 25n, where d is a semi-distance. 
© ap Dj K(Pj||Po) < a log M, with0 <a < 1/8. 


Then 
Equation: 


a Ps (a (3 f) > sn) > ara ya (a i fi) > sn) 
/M 
(te? — >0 


We will use this theorem to show that the estimator of Lecture 4 is optimal. Recall the setup of Lecture 
4. Let 
Equation: 


F ={f = (f(t) — f(s)| < Lit — s\vt, 8} 


i.e. the class of Lipschitz functions with constant D. Let 


Equation: 
CHUh;, = 1,520 
Equation: 
Yi = f (vi) + Wi 
E(W;] =0,E[W?] = 0? < co, Wj, W; are indepedent if i ¥ j. In that lecture, we constructed 


an estimator f, such that 
Equation: 


fa — fF ||] =O (n°? 
sup Ella — fl[']=0 (2) 


Is this the best we can do? 


We are going to construct a collection fo,---, fz € FY and apply Theorem 2. Notice that the metric of 
interest is d cs f) = || fn — f ||, asemi-distance. Let W; “ W (0,07). Letm € N,h =1/mand 
define 

Equation: 


Lh L 
K(z)= (+ =i ellen - al — 2z|T\2\<n/2 


K(x) 
Lh/2 


-h/2 h/2 


Note that |K (a) — K(b)| < L|a — 6|, Va, b. The subclass we are going to consider are functions of the 
form 


f_ w(x) 


AA 


i.e. “bump" functions. Let 2 = {0,1}"" be the collection of binary vectors of length m, e.g. 
w = (1,0,1,---,0) € 2. Define 
Equation: 


Note that for w, w’ € 92, 
Equation: 


——— (f) So (e—uytee(2— Fern)” 
Vow, w") [wae 


where p(w, w’) is the Hamming distance, p(w, w’) = 0724 |wi — w |? = oie, |wi — wi|. Now 
Equation: 


h/2 3 2 
[e@=2/ Pree == e 
0 328. 12 


sO 
Equation: 


d (fos fu’) = yolu,w aah 


Since |2| = 2”, the number of functions in our class is 2”. Turns out, we do not need to consider all 
functions f,,, w € §, but only a select few. Using all the functions leads to a looser lower bound of the 
form n~1, which corresponds to the parametric rate. The problem under consideration is non- 
parametric, and hence we expect a slower rate of convergence. To get a tighter lower bound, the 
following result is of use: 

Lemma 

Varshamov-Gilbert '62 


Let m > 8. There exists a subset fw, vee wi) of Q such that w) = (0,0,---,0), 
Equation: 


p (ww) 2 7 VO <j<k<M and M>2%. 


What this lemma says is that there are many (~ 2”) sequences in (2 that are very different (i.e. 
p(w), w'*)) ~ m). We are going to use the lemma to construct a useful set of hypotheses. Let 
fw, tee wi) \ be the class of sequences in the lemma and define 

Equation: 


A 
fi = fu, j € {0,---,M} 


We now need to look at the conditions of Theorem 2 and choose m appropriately. 


First note that for 7 4 k, 
Equation: 


d (fj, fx) = 4/ p(w), w)) nh? > 


A : 
Now let Pj = Po xd € {0,---, M}. Then 


Equation: 
pi 
Yi,---,Ym 
K(P;||Po) = E,;|log —-™ 
(0) 
PY oN 
n (j)y, n 
= 7 pty 1 Oho: 
i= Py, i=1 


1 SX/Lh\? LP L? 
oe oe. 24 ie > h2 = ee —2 
~ 202 », ( 2 ) 802 . 802 se 


Now notice that log M > = log 2 (from Lemma [link]). We want to choose m such that 


Equation: 
1 i _9 m 
ap 2m ||Po) < ganm™ <a log 2 < a log M 
This gives 
Equation: 


so take m = | Con’? -- a Now 


Equation: 
L 
OS tk) S —_m! > 2const n 1/8 for n > no (const) 
4/6 

Therefore, 
Equation: 

infsup Py (|| fr — f|| > const n-¥/3) >e>0 

fr FEF 
or, 


Equation: 


infsup Ps (\\fn — f\\? > const n-?/) >ce>0 
f, FEF 


or after Markov's inequality, 
Equation: 


infsup Ey [Il fo — f\|?] > e+ const n-?? 
fn FEF 


Therefore, the estimator constructed in class attains the optimal rate of convergence. 


